The widespread use of risk profiling systems by public authorities in law enforcement, social security and migration is incompatible with international human rights law and must be banned, Amnesty International said in a new report.
Risk profiling is the assessment of whether a person or group is likely to break a law or a rule. Governments use it to identify potential offenders before any offence has been committed. These assessments are increasingly being automated based on artificial intelligence techniques and, in many cases, have led to human rights violations. This is the first time that risk profiling has been comprehensively assessed against international human rights standards.
“This report shows how risk profiling systems affects people’s rights and offers scientific and legal arguments that human rights advocates, civil servants, oversight authorities, lawyers and judges can use to contest the use of these systems by states and other entities,” said Alexander Laufer, Amnesty International Netherlands Researcher on technology and human rights.
These systems have for example, been deployed in Denmark’s social benefits fraud investigations, France’s residence permit application system, the Netherlands’ social fraud detection system and Australia’s automated debt recovery scheme.
Risk profiling systems are often cited as a method by which states can streamline services, improve cost-effectiveness, prevent crimes, including fraud, and control migration. These claims, based on the premise of scarce resources, have been debunked as a policy rationale, because they are empirically unsupported and politically useful assumptions that turn poverty and other social issues from political problems into an “efficiency” problem to be solved through automated rationing and surveillance.
Being targeted by risk profiling in high-stakes contexts like law enforcement, social security and immigration causes serious harms. People suffer psychological distress, stigmatization and false accusations of crime, potentially leading to homelessness, deportation, unfair denial of social benefits, or even imprisonment.
Discrimination and intersectional impacts
The report also highlights often overlooked structural and intersectional effects of the discriminatory deployment of risk profiling, even though it is touted by governments as an objective tool. This technology violates the right to equality and non-discrimination and results in discrimination based on race and ethnicity, gender, socio-economic status and disability.
Marginalized groups are the hardest hit. Racialized people, Muslims, people on the move, people with disabilities or chronic illnesses, and people on low incomes are more likely to be labelled as “suspicious,” placed under constant surveillance, and ultimately suffer from harmful decisions.
“This is made worse by the widespread lack of transparency, which leaves people powerless and unable to challenge these systems and decisions that affect their rights. This also poses risks to other human rights, such as the right to fair trial, the presumption of innocence, the right to privacy and data protection, social security and an adequate standard of living and full realization of human dignity,” said Alexander Laufer.
These discriminatory impacts go beyond bias and individual rights. While reducing crime or fraud can be a legitimate reason for the deployment of these systems, they may still be abused to provide cover for invasive community-wide surveillance and result in disproportionate human rights harms.
“The way these tools are deployed can be the result of pre-existing stereotypes and prejudices which regard marginalized groups as inherently criminal or dangerous. Individuals or groups are transformed from statistical, hypothetical suspects into actual suspects, solidifying pre-existing prejudices or generating new ones. This is the result of existing systemic discrimination,” said Alexander Laufer.
Doubts about effectiveness
Amnesty also found that many government processes for building risk profiling systems lack rigorous scientific methods that typically prevent the production of misleading results or their misinterpretation. Governments often describe risk profiling as a cost-effective way to fight crime, social security fraud and irregular migration, but there’s very little scientific evidence to validate those claims. Instead, research shows that these tools are scientifically dubious and consistently inaccurate.
Predicting whether an individual will commit a crime or social security fraud is a scientifically fraught enterprise, the research indicates. The data required to perform the task with sufficient quality and accuracy, does not and will never exist. For example, complex behaviors such as crime or social security fraud are extremely hard to measure, predict and verify. Therefore, unreliable and biased proxies need to be used for predicting these behaviors, such as the use of re-arrest to predict re-offending, or involuntary mistakes in social security instead of fraudulent claims, which put racialized and marginalized communities at risk.
Therefore, the scientific validity of risk profiling models is fundamentally weak and can cause serious harm to targeted people. This, in turn, makes risk profiling hard to justify.
“Attempts to predict fraud or criminality often amount to automatically turning marginalized communities and individuals into suspects rather than evidence-based decision-making,” said Alexander Laufer.
“It is impossible to design an objective or neutral risk profiling algorithm. Data about people is never objective. When governments use past social data to predict who is going to commit a crime such as fraud, they inevitably target individuals who belong to historically oppressed or marginalized groups thereby reproducing and compounding past injustices. There is no technical way to fix societal issues.”
Amnesty International and others have documented case studies showing that risk profiling systems disproportionately associate people experiencing one or multiple forms of discrimination or marginalization with higher criminal or financial risk.
Ban risk profiling
Risk profiling in migration, social security fraud detection and crime control must be banned. Predictive profiling and risk assessment systems, whether data driven or rules-based, should be banned in high stakes contexts even when humans make the final decisions. States must also enact or amend existing laws to enforce this ban. Until such laws are in place, authorities must not use these systems and stop their development, production and sale.
Contact: [email protected]