738 labelers and 7 frontier models are classifying 3,000 pediatric chest radiographs; 235k reads collected so far.
235kOpinions collected78 reads per case, on average
A fractionof in-house review, per caseAll reads and aggregation included
88.0vs 58.9Consensus accuracy vs best model+29.1 points over Gemini 3.1 Pro
Task-level quality
Every graded case, against ground truth
2,995 cases in this contest carry a known correct answer. This is where the accuracy-weighted consensus landed on all of them.
Consensus label ↓ Correct label →
bacterial pneumonia
none
viral pneumonia
Row total
bacterial pneumonia
1,36491.1% of column
20.2%
15938.4%
1,525
none
382.5%
1,07599.2% of column
5713.8%
1,170
viral pneumonia
956.3%
70.6%
19847.8% of column
300
Cases with this answer
1,497
1,084
414
2,995
Consensus accuracy
91.1%
99.2%
47.8%
88.0%
Confusion matrix · Topic 679 · 2,995 graded cases · 2,637 on the diagonal
One of the 1,364
A single case, from the highlighted cell
The matrix says the consensus is right 91% of the time on cases like this one. The replay shows the mechanism that gets it there.
Case 1189372 · Image classification
Identify pneumonia findings
A · bacterial pneumoniaB · noneC · viral pneumonia
24
qualified reads to settle
3
filtered below Q 45
71%
agreement at settle
real recorded reads
Contest statistics from Centaur task 679; confusion matrix over its graded cases. Replay of real opinions from a live Centaur case; timeline compressed for playback, Q-scores as recorded at submission.
