Binary subset
All 1,005 questions · Binary results · Scoring · Download question IDs
These 214 questions from the 1,005-question set supply the post's precision, recall, false-alarm and missed-detection results. Each has two answer choices and an explicit positive label from its source. Fifteen use adapted choice representations.
Each question page shows its source, reference answer and all 13 recorded model outcomes. Positive choice is the choice key treated as positive in scoring; its meaning varies by source. ‘Correct / answered’ counts saved answers across all 13 models. Questions whose text is obtained separately remain listed with their results and recovery instructions.
Retrieval · 16 questions
Concepts and categorization · 18 questions
Physical cognition · 3 questions
| Question | Source | Positive choice | Correct / answered |
|---|---|---|---|
| Recover locally — boolean | NEWTON | 1 |
7/13 |
| Recover locally — boolean | NEWTON | 1 |
10/13 |
| Recover locally — boolean | NEWTON | 1 |
12/13 |
Spatial cognition · 15 questions
| Question | Source | Positive choice | Correct / answered |
|---|---|---|---|
| Is the red square above the pink rectangle? | bAbI | yes |
12/13 |
| Is the triangle above the blue square? | bAbI | yes |
11/13 |
| Is the yellow square below the triangle? | bAbI | yes |
12/13 |
| Is the pink rectangle below the red square? | bAbI | yes |
12/13 |
| Is the red square below the yellow square? | bAbI | yes |
12/13 |
| Is the triangle above the blue square? | bAbI | yes |
11/13 |
| Is the triangle to the right of the pink rectangle? | bAbI | yes |
11/13 |
| Is the red square below the red sphere? | bAbI | yes |
11/13 |
| Is the triangle above the red square? | bAbI | yes |
11/13 |
| Is the yellow square to the left of the pink rectangle? | bAbI | yes |
11/13 |
| Is the red sphere above the yellow square? | bAbI | yes |
10/13 |
| Is the yellow square below the triangle? | bAbI | yes |
11/13 |
| Is the blue square to the left of the red square? | bAbI | yes |
10/13 |
| Is the yellow square below the red sphere? | bAbI | yes |
12/13 |
| Is the red square above the red sphere? | bAbI | yes |
10/13 |
Numerical cognition · 15 questions
Causal reasoning · 57 questions
Uncertainty monitoring · 57 questions
Communication · 29 questions
Rule application · 4 questions
| Question | Source | Positive choice | Correct / answered |
|---|---|---|---|
| Does the target assistant step contain an error under a process-quality evaluation? A correct useful… | AgentProcessBench | yes |
12/13 |
| Does the target assistant step contain an error under a process-quality evaluation? A correct useful… | AgentProcessBench | yes |
11/13 |
| Does the target assistant step contain an error under a process-quality evaluation? A correct useful… | AgentProcessBench | yes |
9/13 |
| Does the target assistant step contain an error under a process-quality evaluation? A correct useful… | AgentProcessBench | yes |
11/13 |