Questions
Each capability links to question pages with sources and all 13 recorded model outcomes. The last column counts questions whose complete text is included in this repository.
The questions come from datasets with different licenses. We include full text where the source terms clearly allow us to share it here. For 148 questions, permission to republish the text is unclear or additional license requirements apply. We therefore provide original download links and recovery tools so readers can obtain those questions from their sources instead.
All 1,005 questions still have their IDs, reference answer keys and recorded model results included. You can verify the reported scores without downloading the missing texts.
Choose a capability below to explore the questions and recorded model answers.
| Capability | Questions | Question texts in this repo |
|---|---|---|
| Retrieval | 67 | 67 |
| State tracking and updating | 67 | 67 |
| Concepts and categorization | 67 | 49 |
| Relations | 67 | 67 |
| Analogy | 67 | 67 |
| Learning and induction | 67 | 34 |
| Physical cognition | 67 | 6 |
| Spatial cognition | 67 | 67 |
| Numerical cognition | 67 | 67 |
| Causal reasoning | 67 | 67 |
| Uncertainty monitoring | 67 | 67 |
| Information seeking | 67 | 67 |
| Social cognition | 67 | 50 |
| Communication | 67 | 67 |
| Rule application | 67 | 48 |
Browse the 214-question binary subset used for precision, recall, false alarms and missed detections.
Sharing decisions for every question · Download released inputs · All saved answers