1,834 labelers and 7 frontier models are classifying 5,731 pediatric chest radiographs; 244k reads collected so far.
244kOpinions collected43 reads per case, on average
A fractionof in-house review, per caseAll reads and aggregation included
74.8vs 58.9All-case accuracy+15.9 points over Gemini 3.1 Pro
Task-level quality
Every consensus case, against ground truth
Of the 5,731 cases in this contest, 3,002 have reached consensus: 70%+ agreement among 3+ qualified opinions. This matrix covers those. The 74.8% above and on the datasets page is scored over every case, including the ones still unsettled, so it runs lower.
Consensus label ↓ Correct label →
bacterial pneumonia
none
viral pneumonia
Row total
bacterial pneumonia
1,36691.2% of column
40.4%
16539.3%
1,535
none
352.3%
1,07699.3% of column
5914.0%
1,170
viral pneumonia
976.5%
40.4%
19646.7% of column
297
Cases with this answer
1,498
1,084
420
3,002
Consensus accuracy
91.2%
99.3%
46.7%
87.9%
Confusion matrix · Topic 679 · 3,002 consensus cases · 2,638 on the diagonal
One of the 1,366
A single case, from the highlighted cell
The matrix says the consensus is right 91% of the time on cases like this one. The replay shows the mechanism that gets it there.
Case 1189372 · Image classification
Identify pneumonia findings
A · bacterial pneumoniaB · noneC · viral pneumonia
24
qualified reads to settle
3
filtered below Q 45
71%
agreement at settle
real recorded reads
Contest statistics from Centaur task 679; confusion matrix over its consensus cases, headline accuracy over all cases. Replay of real opinions from a live Centaur case; timeline compressed for playback, Q-scores as recorded at submission.
