Historical Bias: The COMPAS Case Study

Even accurate data can encode an unjust history. The COMPAS recidivism tool shows what happens when AI learns from a biased past.

35 min · Reviewed 2026

When the Past Is Not Fair

COMPAS is a commercial risk-assessment tool used by US courts to predict whether a defendant will re-offend. In 2016, ProPublica published an investigation showing that COMPAS was nearly twice as likely to falsely flag Black defendants as high-risk compared to white defendants. This case became the most famous example of historical bias in AI.

What the numbers showed

Error type	Black defendants	White defendants
Falsely labeled high-risk (didn't re-offend)	45%	23%
Falsely labeled low-risk (did re-offend)	28%	48%
Overall accuracy	~65%	~65%

Why this happened

COMPAS was trained on historical data: arrests, convictions, re-arrests. But the US criminal justice system has a well-documented history of policing Black neighborhoods more heavily, leading to more arrests for equivalent behavior. The model learned to replicate that historical pattern, not because the programmers intended bias, but because the data itself reflected generations of unequal enforcement.

The fairness impossibility

What we learned

Predictive accuracy is not fairness
Historical data encodes historical injustice
Fairness has multiple, sometimes conflicting definitions
Audit for disparate impact per group, not just overall accuracy
High-stakes predictions (liberty, healthcare, loans) demand extra scrutiny

In 2016, Wisconsin's supreme court ruled that COMPAS could be used in sentencing, but only with warnings about its limitations. The debate continues to this day, and COMPAS remains in use in several US jurisdictions.

The big idea: a model trained on an unjust past will perpetuate that injustice into the future. Technical accuracy is not a defense. AI used in high-stakes decisions demands moral choices, not just statistical ones.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-historical-bias-compas

What is the main idea of "Historical Bias: The COMPAS Case Study"?
1. Even accurate data can encode an unjust history. The COMPAS recidivism tool shows what happens when AI learns from a biased past.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Historical Bias: The COMPAS Case Study"?
1. COMPAS
2. historical bias
3. recidivism
4. algorithmic fairness
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Predictive accuracy is not fairness
4. Treat the AI output as automatically correct
What should a careful learner remember about "The paradox"?
1. Use AI to draft or organize ideas about historical bias, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about historical bias be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about historical bias.
Which action would help you apply "Historical Bias: The COMPAS Case Study" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Historical data encodes historical injustice

← Back to interactive lesson

Tendril · Creators · AI Foundations

Historical Bias: The COMPAS Case Study

Even accurate data can encode an unjust history. The COMPAS recidivism tool shows what happens when AI learns from a biased past.

35 min · Reviewed 2026

When the Past Is Not Fair

What the numbers showed

Error type	Black defendants	White defendants
Falsely labeled high-risk (didn't re-offend)	45%	23%
Falsely labeled low-risk (did re-offend)	28%	48%
Overall accuracy	~65%	~65%

Why this happened

The fairness impossibility

What we learned

Predictive accuracy is not fairness
Historical data encodes historical injustice
Fairness has multiple, sometimes conflicting definitions
Audit for disparate impact per group, not just overall accuracy
High-stakes predictions (liberty, healthcare, loans) demand extra scrutiny

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-historical-bias-compas

What is the main idea of "Historical Bias: The COMPAS Case Study"?
1. Even accurate data can encode an unjust history. The COMPAS recidivism tool shows what happens when AI learns from a biased past.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Historical Bias: The COMPAS Case Study"?
1. COMPAS
2. historical bias
3. recidivism
4. algorithmic fairness
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Predictive accuracy is not fairness
4. Treat the AI output as automatically correct
What should a careful learner remember about "The paradox"?
1. Use AI to draft or organize ideas about historical bias, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about historical bias be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about historical bias.
Which action would help you apply "Historical Bias: The COMPAS Case Study" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Historical data encodes historical injustice

← Back to interactive lesson