Calibration

A calibrated classifier is one whose probability estimates match real frequencies. If it says 70 percent across a batch of predictions, 70 percent of those should be correct. That is an uncommon property in modern LLMs.

The reliability diagram

To check calibration, bin predictions by confidence (0-10 percent, 10-20 percent, ...). For each bin, plot average confidence vs empirical accuracy. A perfectly calibrated model hugs the diagonal. Overconfidence bends the line below; underconfidence bends it above.

Perfect calibration:

Accuracy
 1.0 |          *
 0.8 |       *
 0.6 |    *
 0.4 | *
 0.0 *--------- Confidence
      0.0  0.6  1.0

Real models often look like:

Accuracy
 1.0 |             *
 0.8 |        *      * (overconfident)
 0.6 |   *
 0.4 | *
 0.0 *---------
     Confidence too high relative to truth.

Expected Calibration Error (ECE)

ECE sums the gap between confidence and accuracy across bins, weighted by bin size. Lower is better. A well-calibrated model has ECE below 0.05. A raw out-of-the-box LLM often sits at 0.15 or higher.

Why LLMs miscalibrate

• RLHF training rewards confident responses — hedging sounds weak
• Text never contains explicit probability labels to learn from
• The Softmax output is sensitive to fine-tuning choices
• In-context examples can push confidence up or down arbitrarily

Fixes that help

1. 1Temperature scaling: post-hoc sharpen or flatten the probability curve
2. 2Prompting for probabilities then calibrating offline
3. 3Ensemble over samples (semantic entropy)
4. 4Few-shot examples that demonstrate appropriate uncertainty

“Modern neural networks are not calibrated — a phenomenon that has worsened as accuracy improved.”

The big idea: a confident answer is not a correct answer. Calibration is the bridge between the two.