The everyday statistical mistakes
- Confusing statistical significance with practical importance
- P-hacking by running many tests and only reporting the significant ones
- Ignoring base rates (common in medical / screening contexts)
- Underpowered studies that find 'no effect' when the real problem is sample size
- Treating correlation as causation
- Comparing groups on many variables without correction
The sanity-check prompt
- Run the sanity-check BEFORE you write the results section
- For any 'p < 0.05' headline, also report the effect size and confidence interval
- Ask about base rates when the result involves screening or rare events
- Ask whether the finding could be a coding error
The big idea: LLMs catch basic statistical mistakes fast and free. Run the sanity check on every analysis. If it matters, also ask a human.
End-of-lesson check
15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-research-statistical-sanity-checking-creators
What is the core idea behind "Statistical Sanity-Checking: AI As Your Second Statistician"?
- Before you trust any result — from you or from AI — run a sanity check. LLMs are surprisingly good at catching your mistakes.
- ASReview
- Technology — anything tech-related more than 3 years old is suspect
- Apply design in your research workflow to get better results
Which term best describes a foundational idea in "Statistical Sanity-Checking: AI As Your Second Statistician"?
- effect size
- p-value
- base rate
- power analysis
A learner studying Statistical Sanity-Checking: AI As Your Second Statistician would need to understand which concept?
- p-value
- base rate
- effect size
- power analysis
Which of these is directly relevant to Statistical Sanity-Checking: AI As Your Second Statistician?
- p-value
- effect size
- power analysis
- base rate
Which of the following is a key point about Statistical Sanity-Checking: AI As Your Second Statistician?
- Confusing statistical significance with practical importance
- P-hacking by running many tests and only reporting the significant ones
- Ignoring base rates (common in medical / screening contexts)
- Underpowered studies that find 'no effect' when the real problem is sample size
Which of these does NOT belong in a discussion of Statistical Sanity-Checking: AI As Your Second Statistician?
- P-hacking by running many tests and only reporting the significant ones
- Confusing statistical significance with practical importance
- ASReview
- Ignoring base rates (common in medical / screening contexts)
Which statement is accurate regarding Statistical Sanity-Checking: AI As Your Second Statistician?
- For any 'p < 0.05' headline, also report the effect size and confidence interval
- Ask about base rates when the result involves screening or rare events
- Run the sanity-check BEFORE you write the results section
- Ask whether the finding could be a coding error
Which of these does NOT belong in a discussion of Statistical Sanity-Checking: AI As Your Second Statistician?
- For any 'p < 0.05' headline, also report the effect size and confidence interval
- Ask about base rates when the result involves screening or rare events
- Run the sanity-check BEFORE you write the results section
- ASReview
What is the key insight about "Template" in the context of Statistical Sanity-Checking: AI As Your Second Statistician?
- Here is my analysis: [paste]. Act as a skeptical statistician.
- ASReview
- Technology — anything tech-related more than 3 years old is suspect
- Apply design in your research workflow to get better results
What is the key insight about "LLMs can also be wrong about stats" in the context of Statistical Sanity-Checking: AI As Your Second Statistician?
- ASReview
- LLMs occasionally recommend tests that don't apply to your data, or mis-state assumptions.
- Technology — anything tech-related more than 3 years old is suspect
- Apply design in your research workflow to get better results
What is the key warning about "Maintain methodological rigour" in the context of Statistical Sanity-Checking: AI As Your Second Statistician?
- ASReview
- Technology — anything tech-related more than 3 years old is suspect
- AI-assisted research requires transparent disclosure of tools used, validation of outputs against primary sources, and p…
- Apply design in your research workflow to get better results
Which statement accurately describes an aspect of Statistical Sanity-Checking: AI As Your Second Statistician?
- ASReview
- Technology — anything tech-related more than 3 years old is suspect
- Apply design in your research workflow to get better results
- The big idea: LLMs catch basic statistical mistakes fast and free. Run the sanity check on every analysis. If it matters, also ask a human.
Which best describes the scope of "Statistical Sanity-Checking: AI As Your Second Statistician"?
- It focuses on Before you trust any result — from you or from AI — run a sanity check. LLMs are surprisingly good a
- It is unrelated to research workflows
- It applies only to the opposite beginner tier
- It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about Statistical Sanity-Checking: AI As Your Second Statistician?
- ASReview
- The sanity-check prompt
- Technology — anything tech-related more than 3 years old is suspect
- Apply design in your research workflow to get better results
Which of the following is a concept covered in Statistical Sanity-Checking: AI As Your Second Statistician?
- effect size
- base rate
- p-value
- power analysis
