961 labelers and 7 frontier models estimated the calories in 3,490 meal photographs in 3 days.
402kOpinions collected115 reads per case, on average
$0.05Cost per caseAll reads and aggregation included
85.1vs 74.0Consensus accuracy vs best model+11.1 points over GPT-5.2
Task-level quality
Every graded case, against ground truth
3,442 cases in this contest carry a known correct answer. This is where the accuracy-weighted consensus landed on all of them.
Consensus label ↓ Correct label →
0 to 150 cal
150 to 300 cal
more than 300 cal
Row total
0 to 150 cal
1,20588.5% of column
627.1%
30.2%
1,270
150 to 300 cal
15211.2%
69379.7% of column
16713.8%
1,012
more than 300 cal
40.3%
11513.2%
1,04186.0% of column
1,160
Cases with this answer
1,361
870
1,211
3,442
Consensus accuracy
88.5%
79.7%
86.0%
85.4%
Confusion matrix · Topic 6565 · 3,442 graded cases · 2,939 on the diagonal
One of the 1,041
A single case, from the highlighted cell
The matrix says the consensus is right 86% of the time on cases like this one. The replay shows the mechanism that gets it there.
Case 33530108 · Image classification
How many calories do you think are in this meal?
A · 0 to 150 calB · 150 to 300 calC · more than 300 cal
25
qualified reads to settle
20
filtered below Q 70
80%
agreement at settle
real recorded reads
Contest statistics from Centaur task 6565; confusion matrix over its graded cases. Replay of real opinions from a live Centaur case; timeline compressed for playback, Q-scores as recorded at submission.
