Elo Ratings for AI

Arpad Elo invented his rating system in 1960 for the US Chess Federation. The math is a logistic curve: the probability player A beats player B is a smooth function of their rating difference. 400 points of gap means roughly a 91 percent win rate.

Key properties

• Only the difference matters, not the absolute rating
• Ratings update after every game, scaled by expectation
• Beating a stronger opponent earns more points
• Losses to weaker opponents cost more points
• Over many games, rating converges to a stable estimate

Where Elo breaks for AI

1. 1Skill is not one-dimensional — a model great at coding and bad at poetry cannot be summarized as one number
2. 2Non-transitive preferences exist (A beats B, B beats C, C beats A) and Elo cannot represent them
3. 3Rating inflation as new strong models enter the pool
4. 4Limited ability to compare models that never played each other

“The rating system is not a moral judgment but a best-guess estimate of relative strength.”

The big idea: Elo is a compact, elegant way to rank competitors — but a single number hides a lot. Always look at the interval and the category breakdown.