Lesson 201 of 1570
Statistical Significance and P-Values
P-value is one of the most abused numbers in research. Here is what it actually says — and what it does not. 'Model B is no better than model A.' 'The new prompt does not change user satisfaction.' A low p-value means the boring story would rarely produce data that looks like what you saw.
Lesson map
What this lesson covers
Learning path
The main moves in order
- 1A Number Everyone Quotes, Almost Nobody Understands
- 2p-value
- 3null hypothesis
- 4significance
Concept cluster
Terms to connect while reading
Section 1
A Number Everyone Quotes, Almost Nobody Understands
You see p < 0.05 in papers and headlines constantly. What does it actually mean? Precisely: if the null hypothesis were true, the probability of seeing a result this extreme or more extreme is less than 5 percent.
The null hypothesis
The null hypothesis is the boring story. 'Model B is no better than model A.' 'The new prompt does not change user satisfaction.' A low p-value means the boring story would rarely produce data that looks like what you saw.
Common abuses
- P-hacking: running many tests and reporting the significant ones
- Garden of forking paths: trying many analyses until something 'works'
- Publication bias: significant results get published; non-significant ones do not
- Confusing statistical and practical significance
Compare the options
| Phrase heard | What it actually means |
|---|---|
| 'Statistically significant' | P-value below threshold, under one analysis |
| 'Not statistically significant' | Might mean no effect, or might mean not enough data |
| 'Highly significant (p<0.001)' | Less likely by chance under null — but still not proof |
| 'Effect size' | The number that actually matters |
“The difference between 'significant' and 'not significant' is not itself statistically significant.”
Key terms in this lesson
The big idea: p-values are one weak piece of evidence, often presented as if they were a verdict. Effect size and replication matter more.
End-of-lesson quiz
Check what stuck
15 questions · Score saves to your progress.
Tutor
Curious about “Statistical Significance and P-Values”?
Ask anything about this lesson. I’ll answer using just what you’re reading — short, friendly, grounded.
Progress saved locally in this browser. Sign in to sync across devices.
Related lessons
Keep going
Builders · 30 min
Where Training Data Actually Comes From
You cannot understand modern AI without understanding its diet. Let's map where the data comes from, how it gets cleaned, and what that means.
Builders · 25 min
Correlation vs. Causation
The most famous warning in statistics is also the most ignored. Here is how to actually tell them apart.
Builders · 25 min
Missing Data and How to Spot It
Real datasets have holes. Blank cells, NaN, NULL, -999, and the dreaded empty string. Learning to see them is a core skill.