When AI Decides Something That Matters

AI is now involved in hiring, loans, medical care, and criminal sentencing. Here are the documented cases and the frameworks being built in response.

30 min · Reviewed 2026

The Stakes Went Up Fast

Recommender systems picking the next video are low-stakes — if the model is wrong you watch something you did not want to. Systems that decide who gets a job, a loan, a kidney transplant, or a longer prison sentence are not low-stakes. AI is now involved in all of those, and getting it wrong has changed lives.

Case 1: COMPAS and criminal risk scoring

COMPAS is a tool used in US courts since the early 2010s to predict whether a defendant will commit another crime. A 2016 ProPublica investigation found that Black defendants were almost twice as likely as white defendants to be labeled high-risk when they in fact did not reoffend. The company disputed the analysis. Multiple states still use the tool.

Case 2: Amazon's scrapped hiring model

Amazon spent years building an AI to screen engineering resumes. It was trained on ten years of past hires — which were mostly men. The model learned to penalize resumes that contained the word women's (as in women's chess club) and to downrank graduates of two women's colleges. Amazon killed the project in 2018.

Case 3: healthcare risk prediction

A 2019 Science paper looked at an algorithm used by US hospitals to flag patients for extra care. The model used past healthcare spending as a proxy for medical need. Because Black patients had historically received less care for the same conditions, the model systematically under-flagged Black patients. An estimated 200 million people had been scored by similar systems.

How the frameworks are responding

EU AI Act: classifies hiring, credit, law enforcement, and healthcare as high-risk — mandatory risk management, logging, and human oversight from August 2026
NYC Local Law 144 (2023): requires bias audits for automated hiring tools
Colorado AI Act (2024): consumer notification and impact assessments for high-risk AI
FDA (US): regulatory pathway for AI as a medical device, with post-market monitoring

Compare: automated vs. human decisions

Dimension	Human judge/loan officer	AI system
Speed	Slow	Instant
Consistency	Varies person to person	Consistent within version
Bias	Documented, familiar	Documented, harder to see
Appealability	You can ask why	Often opaque
Scalability	One case at a time	Millions per day

Design questions the field is converging on

Is the AI advising a human or deciding alone? High-stakes systems should advise.
Can the subject see what the model used and why it concluded what it did?
Is there a real appeals process that can overturn the model?
Is the model regularly audited on outcomes by demographic group?
Is performance on the most-affected group disclosed in public documentation?

Weapons of math destruction are opaque, they scale, and they damage.
— Cathy O'Neil

The big idea: when AI touches real lives, the math problems become policy problems. Knowing the canonical cases lets you spot the same patterns when they show up in a new product.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ethics-high-stakes-decisions-builders

What is the core idea behind "When AI Decides Something That Matters"?
1. AI is now involved in hiring, loans, medical care, and criminal sentencing. Here are the documented cases and the frameworks being built in response.
2. Build assessment items beyond multiple choice
3. Everyone makes mistakes — even AI. The fix is to keep learning.
4. Draft withdraw-data options with clear next-step instructions.
Which term best describes a foundational idea in "When AI Decides Something That Matters"?
1. COMPAS
2. algorithmic decision-making
3. proxy variable
4. high-risk AI
A learner studying When AI Decides Something That Matters would need to understand which concept?
1. algorithmic decision-making
2. proxy variable
3. COMPAS
4. high-risk AI
Which of these is directly relevant to When AI Decides Something That Matters?
1. algorithmic decision-making
2. COMPAS
3. high-risk AI
4. proxy variable
Which of the following is a key point about When AI Decides Something That Matters?
1. EU AI Act: classifies hiring, credit, law enforcement, and healthcare as high-risk — mandatory risk …
2. NYC Local Law 144 (2023): requires bias audits for automated hiring tools
3. Colorado AI Act (2024): consumer notification and impact assessments for high-risk AI
4. FDA (US): regulatory pathway for AI as a medical device, with post-market monitoring
Which of these does NOT belong in a discussion of When AI Decides Something That Matters?
1. NYC Local Law 144 (2023): requires bias audits for automated hiring tools
2. Colorado AI Act (2024): consumer notification and impact assessments for high-risk AI
3. Build assessment items beyond multiple choice
4. EU AI Act: classifies hiring, credit, law enforcement, and healthcare as high-risk — mandatory risk …
Which statement is accurate regarding When AI Decides Something That Matters?
1. Can the subject see what the model used and why it concluded what it did?
2. Is there a real appeals process that can overturn the model?
3. Is the AI advising a human or deciding alone? High-stakes systems should advise.
4. Is the model regularly audited on outcomes by demographic group?
Which of these does NOT belong in a discussion of When AI Decides Something That Matters?
1. Is there a real appeals process that can overturn the model?
2. Can the subject see what the model used and why it concluded what it did?
3. Build assessment items beyond multiple choice
4. Is the AI advising a human or deciding alone? High-stakes systems should advise.
What is the key insight about "The pattern" in the context of When AI Decides Something That Matters?
1. In each case, the failure was not a bug. The system did exactly what it was asked to do.
2. Build assessment items beyond multiple choice
3. Everyone makes mistakes — even AI. The fix is to keep learning.
4. Draft withdraw-data options with clear next-step instructions.
What is the key insight about "Human in the loop is not automatic" in the context of When AI Decides Something That Matters?
1. Build assessment items beyond multiple choice
2. Just having a human sign off is not enough. If the human rubber-stamps the AI 99 percent of the time, the AI is effectiv…
3. Everyone makes mistakes — even AI. The fix is to keep learning.
4. Draft withdraw-data options with clear next-step instructions.
What is the recommended tip about "Key insight" in the context of When AI Decides Something That Matters?
1. Build assessment items beyond multiple choice
2. Everyone makes mistakes — even AI. The fix is to keep learning.
3. AI is now involved in hiring, loans, medical care, and criminal sentencing.
4. Draft withdraw-data options with clear next-step instructions.
Which statement accurately describes an aspect of When AI Decides Something That Matters?
1. Build assessment items beyond multiple choice
2. Everyone makes mistakes — even AI. The fix is to keep learning.
3. Draft withdraw-data options with clear next-step instructions.
4. Recommender systems picking the next video are low-stakes — if the model is wrong you watch something you did not want to.
What does working with When AI Decides Something That Matters typically involve?
1. COMPAS is a tool used in US courts since the early 2010s to predict whether a defendant will commit another crime.
2. Build assessment items beyond multiple choice
3. Everyone makes mistakes — even AI. The fix is to keep learning.
4. Draft withdraw-data options with clear next-step instructions.
Which of the following is true about When AI Decides Something That Matters?
1. Build assessment items beyond multiple choice
2. Amazon spent years building an AI to screen engineering resumes. It was trained on ten years of past hires — which were mostly men.
3. Everyone makes mistakes — even AI. The fix is to keep learning.
4. Draft withdraw-data options with clear next-step instructions.
Which best describes the scope of "When AI Decides Something That Matters"?
1. It is unrelated to ethics workflows
2. It applies only to the opposite beginner tier
3. It focuses on AI is now involved in hiring, loans, medical care, and criminal sentencing. Here are the documented
4. It was deprecated in 2024 and no longer relevant

← Back to interactive lesson

Tendril · Builders · Ethics & Society