Lesson 261 of 1570
Catastrophic Risk, Without the Panic
Measured people at serious labs and universities publicly worry about AI going very wrong. Here is what they mean, what they disagree about, and how to read the headlines.
Lesson map
What this lesson covers
Learning path
The main moves in order
- 1The Spectrum, Not the Slogan
- 2catastrophic risk
- 3existential risk
- 4CBRN
Concept cluster
Terms to connect while reading
Section 1
The Spectrum, Not the Slogan
You have seen the headlines: AI could end humanity. You have also seen the opposite: AI doom is sci-fi nonsense. Neither slogan is how working researchers talk. They talk about a spectrum of severity and a set of specific concrete pathways.
The four tracked categories
OpenAI's Preparedness Framework and Anthropic's Responsible Scaling Policy both enumerate similar categories for high-severity risk. In rough agreement:
- 1CBRN: chemical, biological, radiological, nuclear. Can a model meaningfully help a novice build a weapon of mass harm?
- 2Cyber: can a model conduct offensive cyber operations above a specific skill bar?
- 3Model autonomy: can a model operate independently over long horizons, make copies of itself, acquire resources?
- 4Persuasion and manipulation: can a model influence beliefs at scale in ways that destabilize institutions?
What the evidence says today
- CBRN uplift: measurable but limited. 2024-2025 evaluations show models can accelerate research tasks but do not turn a novice into a weapons developer.
- Cyber: frontier models complete apprentice-level offensive security tasks 50% of the time (UK AISI 2025), up from 10% in early 2024. One model passed expert-level tasks typically requiring 10+ years experience.
- Autonomy: METR's time-horizon benchmark shows AI-completable task length doubling every 4-7 months. In late 2025, top models reliably complete ~5-hour tasks.
- Persuasion: hard to measure. Some lab evals were tracked then dropped as benchmarks matured.
Where researchers disagree
- Timing: years vs. decades vs. never for transformative risk
- Mechanism: misuse by humans vs. misaligned AI vs. systemic societal collapse
- Tractability: can alignment keep up with capability, yes or no
- Policy: slow capability, improve safety, or both
Compare the options
| Claim | Honest version | Dishonest version |
|---|---|---|
| AI could help with bioweapons | Measurable uplift on novice tasks; weapons still require wet-lab capability not in the model | AI will design pandemics by Tuesday |
| AI could do long autonomous projects | Task horizons growing exponentially, hours-scale today | AI is about to be CEO |
| AI could destabilize elections | Persuasion and personalized disinfo are cheaper | AI is why your candidate lost |
“I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”
Key terms in this lesson
The big idea: catastrophic risk is a real research agenda with real evidence. The honest version is smaller than the panicky version and bigger than the dismissive version. Read the papers, not the tweets.
End-of-lesson quiz
Check what stuck
15 questions · Score saves to your progress.
Tutor
Curious about “Catastrophic Risk, Without the Panic”?
Ask anything about this lesson. I’ll answer using just what you’re reading — short, friendly, grounded.
Progress saved locally in this browser. Sign in to sync across devices.
Related lessons
Keep going
Builders · 30 min
Copyright and AI: Who Owns What?
Generative AI trained on copyrighted work has triggered the biggest wave of copyright lawsuits in the internet era. Here is the state of the fight.
Builders · 25 min
AI and Homework: Where Is the Honest Line?
Using AI on schoolwork is not simply cheating or not cheating. It depends on the task, the rules, and what you are learning to do. Here is how to think about it.
Builders · 30 min
The EU AI Act in Plain English
The world's most ambitious AI law passed in 2024. Here is what it actually does, when it kicks in, and why it matters if you do not live in Europe.