Each point pairs a stated-confidence bin with the rate actually observed in that bin. A perfectly calibrated analyst would sit on the diagonal: state 70 percent and be right 70 percent of the time. The gap between the two is the expected calibration error (ECE).
A1 — ECE 0.114
stated 0.36 versus 0.00 observed across 4 matches; stated 0.45 versus 0.52 observed across 23 matches; stated 0.55 versus 0.66 observed across 32 matches; stated 0.64 versus 0.81 observed across 27 matches; stated 0.75 versus 0.77 observed across 13 matches; stated 0.86 versus 0.80 observed across 5 matches.
A1-zero — ECE 0.135
stated 0.38 versus 0.00 observed across 2 matches; stated 0.45 versus 0.39 observed across 13 matches; stated 0.55 versus 0.73 observed across 37 matches; stated 0.65 versus 0.74 observed across 23 matches; stated 0.75 versus 0.65 observed across 17 matches; stated 0.84 versus 1.00 observed across 7 matches; stated 0.93 versus 0.80 observed across 5 matches.
A2 — ECE 0.112
stated 0.33 versus 0.33 observed across 6 matches; stated 0.45 versus 0.63 observed across 24 matches; stated 0.54 versus 0.56 observed across 25 matches; stated 0.63 versus 0.82 observed across 34 matches; stated 0.73 versus 0.70 observed across 10 matches; stated 0.83 versus 0.80 observed across 5 matches.
A2-static — ECE 0.136
stated 0.33 versus 0.33 observed across 6 matches; stated 0.45 versus 0.63 observed across 24 matches; stated 0.54 versus 0.50 observed across 26 matches; stated 0.64 versus 0.87 observed across 31 matches; stated 0.74 versus 0.80 observed across 15 matches; stated 0.83 versus 0.50 observed across 2 matches.
A3 — ECE 0.100
stated 0.33 versus 0.33 observed across 6 matches; stated 0.45 versus 0.63 observed across 24 matches; stated 0.55 versus 0.54 observed across 28 matches; stated 0.63 versus 0.84 observed across 25 matches; stated 0.74 versus 0.79 observed across 14 matches; stated 0.84 versus 0.86 observed across 7 matches.
ELO — ECE 0.114
stated 0.39 versus 0.56 observed across 9 matches; stated 0.44 versus 0.48 observed across 44 matches; stated 0.54 versus 0.77 observed across 35 matches; stated 0.64 versus 0.69 observed across 16 matches.
ClosingLine — ECE 0.104
stated 0.37 versus 0.33 observed across 9 matches; stated 0.45 versus 0.65 observed across 20 matches; stated 0.55 versus 0.62 observed across 21 matches; stated 0.64 versus 0.74 observed across 23 matches; stated 0.73 versus 0.85 observed across 13 matches; stated 0.83 versus 0.81 observed across 16 matches; stated 0.92 versus 0.50 observed across 2 matches.
Mo — ECE 0.074
stated 0.53 versus 0.40 observed across 30 matches; stated 0.62 versus 0.63 observed across 40 matches; stated 0.73 versus 0.83 observed across 23 matches; stated 0.83 versus 0.70 observed across 10 matches; stated 0.90 versus 1.00 observed across 1 matches.
Reliability curve
Each point is a confidence band. The horizontal axis is what the analyst said. The vertical axis is what actually happened. The diagonal is perfect calibration: a point above it means the analyst was underconfident, below means overconfident.
The race to the best architecture
The whole experiment is here. Each line is one AI architecture's calibration error as matches accumulate. Lower is better, so the line that settles toward the bottom is the architecture we would build. The three AI systems are bold; the betting market, ELO, and Mo run faint as reference.
The analysts, side by side
This table is the control for all three charts above.
When an analyst says 70 percent, does that pick win 70 percent of the time? That is what this chart answers (statisticians call it a reliability curve). Updated after every completed match. It appears after the first scored matchday.
Live after one group matchday
0 of ~24 scored matches. The reliability curve goes live at 24, one full group matchday.
The reliability curve goes live once the first group matchday is complete. A curve needs about 24 scored matches before it says something honest. Showing it earlier would be noise dressed up as signal. Every match until then is already being scored: see the verdict on any finished match page.
