Confidence Intervals

When a model scores 72 percent on a benchmark, that is a point estimate. The real question is: what range of values is plausible? That range is the confidence interval.

What 95% CI actually means

A 95% confidence interval is an interval built by a procedure that, in the long run, captures the true value 95 percent of the time. '72%, 95% CI [68, 76]' means roughly: if you repeated the experiment many times, 95% of the computed intervals would contain the true score.

Estimating CIs quickly

1. 1For a percentage: CI ≈ score ± 1.96 × sqrt(p(1-p)/n)
2. 2For the mean: CI ≈ mean ± 1.96 × std/sqrt(n)
3. 3For anything messy: bootstrap (resample and recompute many times)

# Bootstrap CI for model accuracy
import numpy as np
results = [...]  # 1 for correct, 0 for wrong
boots = []
for _ in range(10000):
    sample = np.random.choice(results, size=len(results), replace=True)
    boots.append(np.mean(sample))
low, high = np.percentile(boots, [2.5, 97.5])
print(f"95% CI: [{low:.3f}, {high:.3f}]")

“A point estimate without an interval is a guess that forgot to mention its uncertainty.”

The big idea: every number has a halo of uncertainty around it. Always ask for the halo.