Historical Bias: The COMPAS Case Study

COMPAS is a commercial risk-assessment tool used by US courts to predict whether a defendant will re-offend. In 2016, ProPublica published an investigation showing that COMPAS was nearly twice as likely to falsely flag Black defendants as high-risk compared to white defendants. This case became the most famous example of historical bias in AI.

What the numbers showed

Why this happened

COMPAS was trained on historical data: arrests, convictions, re-arrests. But the US criminal justice system has a well-documented history of policing Black neighborhoods more heavily, leading to more arrests for equivalent behavior. The model learned to replicate that historical pattern, not because the programmers intended bias, but because the data itself reflected generations of unequal enforcement.

The fairness impossibility

What we learned

• Predictive accuracy is not fairness
• Historical data encodes historical injustice
• Fairness has multiple, sometimes conflicting definitions
• Audit for disparate impact per group, not just overall accuracy
• High-stakes predictions (liberty, healthcare, loans) demand extra scrutiny

In 2016, Wisconsin's supreme court ruled that COMPAS could be used in sentencing, but only with warnings about its limitations. The debate continues to this day, and COMPAS remains in use in several US jurisdictions.

The big idea: a model trained on an unjust past will perpetuate that injustice into the future. Technical accuracy is not a defense. AI used in high-stakes decisions demands moral choices, not just statistical ones.