Lesson 359 of 2116
Statistical Sanity-Checking: AI As Your Second Statistician
Before you trust any result — from you or from AI — run a sanity check. LLMs are surprisingly good at catching your mistakes.
Lesson map
What this lesson covers
Learning path
The main moves in order
- 1The everyday statistical mistakes
- 2sanity check
- 3effect size
- 4p-value
Concept cluster
Terms to connect while reading
Section 1
The everyday statistical mistakes
- Confusing statistical significance with practical importance
- P-hacking by running many tests and only reporting the significant ones
- Ignoring base rates (common in medical / screening contexts)
- Underpowered studies that find 'no effect' when the real problem is sample size
- Treating correlation as causation
- Comparing groups on many variables without correction
The sanity-check prompt
- 1Run the sanity-check BEFORE you write the results section
- 2For any 'p < 0.05' headline, also report the effect size and confidence interval
- 3Ask about base rates when the result involves screening or rare events
- 4Ask whether the finding could be a coding error
Key terms in this lesson
The big idea: LLMs catch basic statistical mistakes fast and free. Run the sanity check on every analysis. If it matters, also ask a human.
End-of-lesson quiz
Check what stuck
15 questions · Score saves to your progress.
Tutor
Curious about “Statistical Sanity-Checking: AI As Your Second Statistician”?
Ask anything about this lesson. I’ll answer using just what you’re reading — short, friendly, grounded.
Progress saved locally in this browser. Sign in to sync across devices.
Related lessons
Keep going
Creators · 11 min
AI Power-Analysis Narrative: Drafting Sample-Size Justification Sections
AI can draft power-analysis sample-size justification narratives, but the effect-size assumption stays with the investigator.
Creators · 11 min
Meta-Analysis Assistance: Where AI Helps And Where It Must Not
Meta-analysis demands precision. AI can accelerate extraction and screening — but the effect-size calculations must stay under human control.
Creators · 40 min
Literature Review With LLMs: Scope First, Search Second
Use an LLM to define the scope of your lit review before touching a search engine — the single highest-leverage move in modern research workflow.