Running Your Own Small Experiment

The best way to truly understand an AI claim is to try it yourself. Here is how to run a small experiment that actually teaches you something.

45 min · Reviewed 2026

Be a Scientist, Not Just a Reader

You do not need a GPU cluster to do AI research. The best small experiments are tiny, specific, and fast — a clear question answered in a Jupyter notebook over an afternoon. Real understanding comes from running them.

A 7-step experiment recipe

Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
Write down what you expect to see before you start
Build the smallest possible test set (30-50 items)
Pick two or three models to compare
Run the test, capture raw outputs, do not over-engineer
Grade with a rubric — LLM-as-judge or human
Write up the result in one page, including what surprised you

Example: does CoT actually help on easy math?

Wei et al. (2022) showed chain-of-thought helps on hard problems. Does it help on grade-school problems too? Run 30 single-digit addition problems, once with 'think step by step' and once without, on the same model. Compare. If you see no difference, you have replicated a real published finding (CoT doesn't help easy tasks). That is real science.

# Tiny experiment skeleton
import anthropic
client = anthropic.Anthropic()

problems = [
    ("2 + 3", "5"),
    ("7 + 8", "15"),
    # ... 28 more
]

def run(prompt_prefix):
    correct = 0
    for q, a in problems:
        resp = client.messages.create(
            model="claude-opus-4-7",
            max_tokens=256,
            messages=[{"role":"user","content":prompt_prefix+q}]
        )
        if a in resp.content[0].text:
            correct += 1
    return correct / len(problems)

print("Plain:", run(""))
print("CoT:", run("Think step by step. "))An afternoon's worth of real AI research, in 20 lines

What counts as a good experiment

One variable changes at a time
Sample size is at least ~30 for any quantitative claim
Results are recorded with raw outputs, not just summary numbers
You wrote your prediction BEFORE running — that keeps you honest
You can describe the limits in one paragraph

The most exciting phrase to hear in science is not 'Eureka!' but 'That's funny...'
— Attributed to Isaac Asimov

The big idea: a one-afternoon experiment teaches you more about AI than a month of reading. Pick a question, run the test, write it up. Repeat.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-run-your-own-experiment

What is the core idea behind "Running Your Own Small Experiment"?
1. The best way to truly understand an AI claim is to try it yourself. Here is how to run a small experiment that actually teaches you something.
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which term best describes a foundational idea in "Running Your Own Small Experiment"?
1. hypothesis
2. experiment
3. pre-registration
4. negative result
A learner studying Running Your Own Small Experiment would need to understand which concept?
1. experiment
2. pre-registration
3. hypothesis
4. negative result
Which of these is directly relevant to Running Your Own Small Experiment?
1. experiment
2. hypothesis
3. negative result
4. pre-registration
Which of the following is a key point about Running Your Own Small Experiment?
1. Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
2. Write down what you expect to see before you start
3. Build the smallest possible test set (30-50 items)
4. Pick two or three models to compare
Which of these does NOT belong in a discussion of Running Your Own Small Experiment?
1. Build the smallest possible test set (30-50 items)
2. Prompt injection: external content overrides system instructions
3. Write down what you expect to see before you start
4. Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
Which statement is accurate regarding Running Your Own Small Experiment?
1. Sample size is at least ~30 for any quantitative claim
2. Results are recorded with raw outputs, not just summary numbers
3. One variable changes at a time
4. You wrote your prediction BEFORE running — that keeps you honest
Which of these does NOT belong in a discussion of Running Your Own Small Experiment?
1. Sample size is at least ~30 for any quantitative claim
2. Prompt injection: external content overrides system instructions
3. One variable changes at a time
4. Results are recorded with raw outputs, not just summary numbers
What is the key insight about "Negative results are results" in the context of Running Your Own Small Experiment?
1. If your hypothesis fails, you still learned something — often more than if it had worked. Write up negative results.
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
What is the key insight about "Cost discipline" in the context of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. It is easy to accidentally spend $40 running 'one more variant.' Set a cost cap before you start.
3. Save two papers a week for a real read
4. Temperature settings tuned per model
What is the recommended tip about "Ground your practice in fundamentals" in the context of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. Save two papers a week for a real read
3. Every AI capability has an underlying mechanism. Understanding that mechanism tells you where it'll fail — which is more…
4. Temperature settings tuned per model
Which statement accurately describes an aspect of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. Save two papers a week for a real read
3. Temperature settings tuned per model
4. You do not need a GPU cluster to do AI research. The best small experiments are tiny, specific, and fast — a clear question answered in a Ju…
What does working with Running Your Own Small Experiment typically involve?
1. Wei et al. (2022) showed chain-of-thought helps on hard problems. Does it help on grade-school problems too? Run 30 single-digit addition pr…
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which of the following is true about Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. The big idea: a one-afternoon experiment teaches you more about AI than a month of reading. Pick a question, run the test, write it up.
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which best describes the scope of "Running Your Own Small Experiment"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It focuses on The best way to truly understand an AI claim is to try it yourself. Here is how to run a small exper
4. It was deprecated in 2024 and no longer relevant

← Back to interactive lesson

Tendril · Creators · AI Foundations

Running Your Own Small Experiment

The best way to truly understand an AI claim is to try it yourself. Here is how to run a small experiment that actually teaches you something.

45 min · Reviewed 2026

Be a Scientist, Not Just a Reader

A 7-step experiment recipe

Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
Write down what you expect to see before you start
Build the smallest possible test set (30-50 items)
Pick two or three models to compare
Run the test, capture raw outputs, do not over-engineer
Grade with a rubric — LLM-as-judge or human
Write up the result in one page, including what surprised you

Example: does CoT actually help on easy math?

# Tiny experiment skeleton
import anthropic
client = anthropic.Anthropic()

problems = [
    ("2 + 3", "5"),
    ("7 + 8", "15"),
    # ... 28 more
]

def run(prompt_prefix):
    correct = 0
    for q, a in problems:
        resp = client.messages.create(
            model="claude-opus-4-7",
            max_tokens=256,
            messages=[{"role":"user","content":prompt_prefix+q}]
        )
        if a in resp.content[0].text:
            correct += 1
    return correct / len(problems)

print("Plain:", run(""))
print("CoT:", run("Think step by step. "))An afternoon's worth of real AI research, in 20 lines

What counts as a good experiment

One variable changes at a time
Sample size is at least ~30 for any quantitative claim
Results are recorded with raw outputs, not just summary numbers
You wrote your prediction BEFORE running — that keeps you honest
You can describe the limits in one paragraph

The most exciting phrase to hear in science is not 'Eureka!' but 'That's funny...'
— Attributed to Isaac Asimov

The big idea: a one-afternoon experiment teaches you more about AI than a month of reading. Pick a question, run the test, write it up. Repeat.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-run-your-own-experiment

What is the core idea behind "Running Your Own Small Experiment"?
1. The best way to truly understand an AI claim is to try it yourself. Here is how to run a small experiment that actually teaches you something.
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which term best describes a foundational idea in "Running Your Own Small Experiment"?
1. hypothesis
2. experiment
3. pre-registration
4. negative result
A learner studying Running Your Own Small Experiment would need to understand which concept?
1. experiment
2. pre-registration
3. hypothesis
4. negative result
Which of these is directly relevant to Running Your Own Small Experiment?
1. experiment
2. hypothesis
3. negative result
4. pre-registration
Which of the following is a key point about Running Your Own Small Experiment?
1. Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
2. Write down what you expect to see before you start
3. Build the smallest possible test set (30-50 items)
4. Pick two or three models to compare
Which of these does NOT belong in a discussion of Running Your Own Small Experiment?
1. Build the smallest possible test set (30-50 items)
2. Prompt injection: external content overrides system instructions
3. Write down what you expect to see before you start
4. Pick one specific hypothesis (e.g., 'Claude beats GPT on Finnish translation')
Which statement is accurate regarding Running Your Own Small Experiment?
1. Sample size is at least ~30 for any quantitative claim
2. Results are recorded with raw outputs, not just summary numbers
3. One variable changes at a time
4. You wrote your prediction BEFORE running — that keeps you honest
Which of these does NOT belong in a discussion of Running Your Own Small Experiment?
1. Sample size is at least ~30 for any quantitative claim
2. Prompt injection: external content overrides system instructions
3. One variable changes at a time
4. Results are recorded with raw outputs, not just summary numbers
What is the key insight about "Negative results are results" in the context of Running Your Own Small Experiment?
1. If your hypothesis fails, you still learned something — often more than if it had worked. Write up negative results.
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
What is the key insight about "Cost discipline" in the context of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. It is easy to accidentally spend $40 running 'one more variant.' Set a cost cap before you start.
3. Save two papers a week for a real read
4. Temperature settings tuned per model
What is the recommended tip about "Ground your practice in fundamentals" in the context of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. Save two papers a week for a real read
3. Every AI capability has an underlying mechanism. Understanding that mechanism tells you where it'll fail — which is more…
4. Temperature settings tuned per model
Which statement accurately describes an aspect of Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. Save two papers a week for a real read
3. Temperature settings tuned per model
4. You do not need a GPU cluster to do AI research. The best small experiments are tiny, specific, and fast — a clear question answered in a Ju…
What does working with Running Your Own Small Experiment typically involve?
1. Wei et al. (2022) showed chain-of-thought helps on hard problems. Does it help on grade-school problems too? Run 30 single-digit addition pr…
2. Prompt injection: external content overrides system instructions
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which of the following is true about Running Your Own Small Experiment?
1. Prompt injection: external content overrides system instructions
2. The big idea: a one-afternoon experiment teaches you more about AI than a month of reading. Pick a question, run the test, write it up.
3. Save two papers a week for a real read
4. Temperature settings tuned per model
Which best describes the scope of "Running Your Own Small Experiment"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It focuses on The best way to truly understand an AI claim is to try it yourself. Here is how to run a small exper
4. It was deprecated in 2024 and no longer relevant

← Back to interactive lesson