Kahlain, Mohamed

About this project

What this is, in plain words.

Q: What is calibration, and why does it matter more than accuracy?

Calibration measures whether a stated confidence can be trusted: when an analyst says 70 percent, does that outcome happen about 70 percent of the time? A model can be accurate yet badly calibrated (overconfident), or modestly accurate yet honest about its uncertainty. inocta-bench treats calibration, not raw accuracy, as the headline finding, scored with expected calibration error (ECE) and the Brier score.

Q: How are the eight analysts scored?

Every analyst publishes a winner pick and a confidence for each match about 75 minutes before kickoff. After the result, predictions are scored on the Brier score and expected calibration error (ECE), with reliability curves and bootstrap 95 percent confidence intervals. Lower Brier and lower ECE are better. Every miss is published.

Q: What are the three AI architectures?

Solo (one AI reasoning over a full match dossier), Pipeline (a specialist chain with a learning loop), and Council (three AIs debate and an anonymous fourth decides). Each has a stripped-down control twin, plus an ELO statistician, the betting-market closing line, and one human, for eight analysts in total.

Q: Is the methodology pre-registered?

Yes. The prediction prompts, model pins, scoring rules and seeded-pattern controls were pre-registered on OSF before the first scored match (DOI 10.17605/OSF.IO/R5SBJ, CC-BY 4.0). Any change after registration is a formal OSF amendment, so the rules cannot be moved to fit the results.

Q: When are predictions published, and can they be changed?

Predictions lock and publish about 75 minutes before kickoff, the betting-market closing-line horizon. Once published they are on the record and never edited. Match pages show a countdown before the publish window.

The finding: the Council architecture recorded the lowest expected calibration error (ECE 0.100 over 104 matches) of the three AI reasoning shapes tested, the best-calibrated AI for business forecasting.

Read the published study →

The short version

We built three AI systems that predict football. During the World Cup they call every game in public, before kickoff, against the betting market, a math rating and one human. Predictions, misses, dollars spent: all published.

Why football, when our day job is AI operations

Football is the test track, not the point. A World Cup hands us 104 decisions in five weeks, each with a hard deadline and a clear answer. Sounds a lot like running a business. What we care about: when an AI says 70 percent, can you trust it? Companies already run real decisions on numbers like that.

The three AI teams

Solo works alone, one pass. Pipeline splits the job into steps and remembers its past mistakes. Council makes three different AIs argue, then a fourth settles it. Everything else is identical. The architecture is the experiment.

The human in the race

Mo, the founder, picks every match too. Pure gut, no data, locked before kickoff. On one match, instinct can beat anything. Across 104? That is the bet, and the answer gets published either way.

What we expect to find

Honestly, we expect nobody to beat the betting market on accuracy. The real contest is calibration: say 70 percent, win 70 percent of the time. That is the number that tells you what an AI can be trusted to decide. It is the same number we watch inside real businesses.

Why you can trust the numbers

The rules went public before the first scored match, so the goalposts could not move. Models stayed frozen all tournament. Misses got the same spotlight as hits, costs show to the cent, and the full dataset is open on OSF alongside the paper.

Who is behind it

inocta.io, AI operations experts out of Toronto and Montréal. We put AI to work inside real businesses, and this benchmark is our method in public: measure before you trust, understand before you automate. You can't automate what you don't understand.

The eight, in one table

Character	Real name	In plain words	What we learn from it
The Soloist	Solo	One AI reads the full match dossier and calls it alone.	Whether one strong model with good data is all you need.
The Purist	Solo-Zero	The same AI with the dossier taken away. It predicts from memory alone.	The gap between Solo and Solo-Zero shows what the match data is actually worth.
The Assembly Line	Pipeline	A chain of specialists: one reads the numbers, one reads everything else, one writes the final call. It keeps notes on its own past misses.	Whether splitting the work into steps, plus learning from mistakes, beats one model working alone.
The Creature of Habit	Pipeline-Static	The exact same chain with the memory of past misses switched off.	The gap between Pipeline and Pipeline-Static shows whether the learning is real or just a story.
The Council	Council	Three different AIs each give an opinion, sometimes disagreeing sharply, and a fourth merges them into one call.	Whether a debate between different AIs beats any single one of them, and whether it is worth the extra cost.
The Statistician	ELO	A pure math rating built from four years of results, like a chess ranking, plus a home-field bump. No AI.	If the AI teams cannot beat simple math, the AI is not adding anything.
The Market	Market	The betting odds at kickoff, turned into percentages. The AIs never see them.	The toughest score in sports. How close anyone gets to the market is the real measure.
The Human	Mo	The founder picks every match on pure gut feel, no data, before kickoff.	Whether human instinct survives 104 matches against machines and math.

Back to the live scoreboard →