961 labelers and 7 frontier models estimated the calories in 3,490 meal photographs in 3 days.
402kOpinions collected115 reads per case, on average
A fractionof in-house review, per caseAll reads and aggregation included
85.0vs 74.0All-case accuracy+11.0 points over GPT-5.2
Task-level quality
Every consensus case, against ground truth
Of the 3,490 cases in this contest, 3,442 have reached consensus: 80%+ agreement among 3+ qualified opinions. This matrix covers those. The 85.0% above and on the datasets page is scored over every case, including the ones that never settled, so it runs lower.
Consensus label ↓ Correct label →
0 to 150 cal
150 to 300 cal
more than 300 cal
Row total
0 to 150 cal
1,20588.5% of column
627.1%
30.2%
1,270
150 to 300 cal
15211.2%
69379.7% of column
16713.8%
1,012
more than 300 cal
40.3%
11513.2%
1,04186.0% of column
1,160
Cases with this answer
1,361
870
1,211
3,442
Consensus accuracy
88.5%
79.7%
86.0%
85.4%
Confusion matrix · Topic 6565 · 3,442 consensus cases · 2,939 on the diagonal
One of the 1,041
A single case, from the highlighted cell
The matrix says the consensus is right 86% of the time on cases like this one. The replay shows the mechanism that gets it there.
Case 33530108 · Image classification
How many calories do you think are in this meal?
A · 0 to 150 calB · 150 to 300 calC · more than 300 cal
25
qualified reads to settle
20
filtered below Q 70
80%
agreement at settle
real recorded reads
Contest statistics from Centaur task 6565; confusion matrix over its consensus cases, headline accuracy over all cases. Replay of real opinions from a live Centaur case; timeline compressed for playback, Q-scores as recorded at submission.
