Automating Suspicion: Risk Profiling as a Smokescreen for Structural Discrimination and Inequality

This report builds on more than a decade of previous research by Amnesty International on the use of digital systems, artificial intelligence (AI), and risk profiling systems in the public sector of European countries, as well as research by other civil society organizations (CSOs) in other parts of the world.

This body of research illustrates how the use of risk profiling systems across a wide range of contexts has consistently resulted in human rights abuses, particularly in high-stakes contexts, defined as the law enforcement, social security and migration policy domains. Amnesty International defines risk profiling as an assessment, evaluation or calculation (sometimes called “prediction”) of the likelihood that individuals or groups will violate a law or rule.

This report provides an overview of these systemic issues, a framing and argumentation to support further investigative work on real-world cases, and provides rights holders, human rights advocates, civil servants, oversight authorities, lawyers and judges with trustworthy scientific and legal arguments to contest the use of risk profiling by states and other entities in high-stakes contexts.

A complete analysis and full understanding of the problems plaguing risk profiling algorithms that often result in human rights violations require insights from several academic disciplines, as well as situating the technology in its historical and social context.

Further, the report explores the often overlooked structural and intersectional effects of discriminatory risk profiling and illustrates how technology can perpetuate discrimination and inequality by providing states with a false veneer of objectivity. We further explore some of the most commonly proposed solutions to the issue of discriminatory profiling and highlight their limits.

Finally, the report brings all the findings together in a legal analysis and bridges the gap between the academic literature and international human rights law (IHRL).