Statistical Significance and P-Values

P-value is one of the most abused numbers in research. Here is what it actually says — and what it does not. 'Model B is no better than model A.' 'The new prompt does not change user satisfaction.' A low p-value means the boring story would rarely produce data that looks like what you saw.

28 min · Reviewed 2026

A Number Everyone Quotes, Almost Nobody Understands

You see p < 0.05 in papers and headlines constantly. What does it actually mean? Precisely: if the null hypothesis were true, the probability of seeing a result this extreme or more extreme is less than 5 percent.

The null hypothesis

The null hypothesis is the boring story. 'Model B is no better than model A.' 'The new prompt does not change user satisfaction.' A low p-value means the boring story would rarely produce data that looks like what you saw.

Common abuses

P-hacking: running many tests and reporting the significant ones
Garden of forking paths: trying many analyses until something 'works'
Publication bias: significant results get published; non-significant ones do not
Confusing statistical and practical significance

Phrase heard	What it actually means
'Statistically significant'	P-value below threshold, under one analysis
'Not statistically significant'	Might mean no effect, or might mean not enough data
'Highly significant (p<0.001)'	Less likely by chance under null — but still not proof
'Effect size'	The number that actually matters

The difference between 'significant' and 'not significant' is not itself statistically significant.
— Gelman and Stern (2006)

The big idea: p-values are one weak piece of evidence, often presented as if they were a verdict. Effect size and replication matter more.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-statistical-significance

What is the main idea of "Statistical Significance and P-Values"?
1. P-value is one of the most abused numbers in research.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Statistical Significance and P-Values"?
1. null hypothesis
2. p-value
3. significance
4. Type I error
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. P-hacking: running many tests and reporting the significant ones
4. Use the first answer without checking it
What should a careful learner remember about "What p-value is not"?
1. Use AI to draft or organize ideas about p-value, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about p-value be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about p-value.
Which action would help you apply "Statistical Significance and P-Values" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. Garden of forking paths: trying many analyses until something 'works'

← Back to interactive lesson

Tendril · Builders · AI Foundations

Statistical Significance and P-Values

28 min · Reviewed 2026

A Number Everyone Quotes, Almost Nobody Understands

The null hypothesis

Common abuses

P-hacking: running many tests and reporting the significant ones
Garden of forking paths: trying many analyses until something 'works'
Publication bias: significant results get published; non-significant ones do not
Confusing statistical and practical significance

Phrase heard	What it actually means
'Statistically significant'	P-value below threshold, under one analysis
'Not statistically significant'	Might mean no effect, or might mean not enough data
'Highly significant (p<0.001)'	Less likely by chance under null — but still not proof
'Effect size'	The number that actually matters

The difference between 'significant' and 'not significant' is not itself statistically significant.
— Gelman and Stern (2006)

The big idea: p-values are one weak piece of evidence, often presented as if they were a verdict. Effect size and replication matter more.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-statistical-significance

What is the main idea of "Statistical Significance and P-Values"?
1. P-value is one of the most abused numbers in research.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Statistical Significance and P-Values"?
1. null hypothesis
2. p-value
3. significance
4. Type I error
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. P-hacking: running many tests and reporting the significant ones
4. Use the first answer without checking it
What should a careful learner remember about "What p-value is not"?
1. Use AI to draft or organize ideas about p-value, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about p-value be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about p-value.
Which action would help you apply "Statistical Significance and P-Values" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. Garden of forking paths: trying many analyses until something 'works'

← Back to interactive lesson