Source terms
Each dataset’s terms still apply. In the table below, Included means we provide the question text. Recover locally means you download the original files and use our tools to reconstruct the questions we used. We checked the source terms on November 9, 2026.
The collection keeps all 1,005 original-question metadata records and all 13,065 baseline outcomes. Full inputs are bundled for 857 questions; the other 148 can be reconstructed locally from original source files under the source terms. Recovered question texts stay in the ignored .local/ folder and are not redistributed with the repository.
| Source | Questions | Question text | Terms | Decision and remaining conditions |
|---|---|---|---|---|
| AG News | 6 | Recover locally | Not established | No explicit redistribution grant for the original news excerpts was established. Obtain the selected text from the original source under its applicable terms. |
| AgentProcessBench | 4 | Included | MIT | The dataset page linked by the official repository declares MIT for the dataset. All four selected trajectories are from tau2, whose original repository also provides an MIT license. Retain both publisher attribution and the original Sierra Research notice. |
| BIG-bench Analogical Similarity | 67 | Included | Apache-2.0 | The pinned BIG-bench repository provides Apache-2.0 terms. Retain the repository notice and task-specific provenance. |
| bAbI | 72 | Included | CC-BY-3.0 | The downloaded original archive contains LICENSE.txt naming Facebook and CC BY 3.0. Retain that notice and mark the multiple-choice adaptations. |
| Banking77 | 7 | Included | CC-BY-4.0 | The pinned publisher dataset card explicitly declares CC BY 4.0. Retain source attribution and normalization notes. |
| BIG-Bench Hard selected tasks | 89 | Included | MIT AND Apache-2.0 | The pinned BBH repository has an MIT license; it selects tasks from BIG-bench. Retain both the BBH MIT notice and the upstream BIG-bench Apache-2.0 notice, plus task identifiers. |
| BigToM | 17 | Included | MIT | The original pinned repository includes an MIT license and author copyright notice. Retain it for the selected synthetic scenarios. |
| BoolQ | 17 | Included | CC-BY-SA-3.0 | The 17 passage/question pairs include original Wikipedia article and contributor-history links, creator credit, normalization notices and CC BY-SA 3.0 terms. The source files match the original Google release checksums. |
| CLadder | 72 | Included | MIT | The pinned original repository has an MIT license. Retain its notice and source IDs. |
| CLINC150 | 7 | Included | CC-BY-3.0 | The pinned source LICENSE is CC BY 3.0. Retain source credit and describe normalization. |
| CLUTRR | 33 | Included | CC-BY-NC-4.0 | Included for the stated noncommercial research release under the original CLUTRR CC BY-NC 4.0 license. Retain creator credit, the license notice and normalization disclosures; the mirror card alone is not the licensing basis. |
| ContractNLI | 16 | Included | CC-BY-4.0 | The original dataset LICENSE and extracted upstream README specify CC BY 4.0. Retain credit, the license, and the original document/question identifiers. |
| CRASS | 10 | Included | Apache-2.0 | The pinned BIG-bench source provides Apache-2.0 terms; retain its notice and the crass_ai task provenance. |
| CRUXEval | 19 | Included | MIT | The pinned README explicitly says CRUXEval is MIT licensed; its LICENSE includes Meta's copyright notice. Retain both source credit and the adaptation notice. |
| EmoWOZ | 6 | Included | CC-BY-NC-4.0 | Included for the stated noncommercial research release under the publisher-declared CC BY-NC 4.0 terms. Retain creator credit, the license notice and normalization disclosures. |
| FANToM | 17 | Recover locally | MIT (project code) | The project code is MIT licensed, but redistribution terms for the separately hosted dataset archive are not explicit. Obtain the 17 dialogues from the publisher and preserve its evaluation-only intended-use statement. |
| FinQA | 10 | Included | MIT | The pinned original repository LICENSE is MIT. Retain its notice and describe the choice-format adaptation; no claim of independent rights over other SEC material is added. |
| HLE-MultiChoice | 7 | Included | MIT | The pinned publisher dataset card declares MIT. Retain publisher attribution and the dataset-card license declaration. |
| iSarcasmEval | 29 | Included | MIT | The pinned original repository provides an MIT license covering its distributed material; retain the original copyright notice. No claim of public-domain status is made for the messages. |
| List Functions | 34 | Included | Apache-2.0 | All 34 selected examples were matched directly to Apache-licensed BIG-bench task files, following the upstream author’s source explanation. The complete seen example pairs and evaluated test input/output values match. Retain the BIG-bench notice and per-question upstream locators. |
| MiniSCAN | 33 | Recover locally | Not established | These questions use a pseudoword adaptation whose exact source files have no established redistribution grant. Obtain those original files for local recovery; another task version or mirror’s license does not establish terms for this version. |
| MMLU | 28 | Included | MIT | The pinned publisher card explicitly declares MIT and links the original benchmark license. Retain attribution and identify normalized multiple-choice labels. |
| MNLI | 6 | Included | OANC (selected non-fiction genres) | The six selected pairs use slate, telephone, oup, verbatim, government and letters, all documented OANC genres. Include the OANC license and Appendix I credits, per-question genre/pair identifiers and source links, and transparent-modification attribution/date. No fiction entries are in this selection. |
| NEWTON | 31 | Recover locally | Not established | No explicit dataset redistribution license was established for the selected source files. Obtain them from the publisher for local recovery; the README’s illustrative LICENSE entry is not a license grant. |
| PIQA | 30 | Recover locally | AFL-3.0 | PIQA uses AFL 3.0, including a recipient-assent condition. The recovery tool requires license acceptance, downloads the original files and verifies the 30 selected inputs in local storage. |
| ProofWriter | 19 | Recover locally | Not established | The original ProofWriter archive and paper provide no dataset license. The separate RuleTaker repository’s Apache license does not explicitly cover this later ProofWriter release; conflicting mirror labels remain insufficient. |
| PUB | 31 | Included | MIT | The pinned publisher dataset card declares MIT. Retain that declaration, the publisher attribution, and task-specific source locators. |
| QuestBench (Google DeepMind) | 38 | Included | CC-BY-4.0 | The original archive LICENSE distinguishes Apache-2.0 software from CC BY 4.0 other materials. The selected questions are dataset entries; retain Google copyright and the archive notice. |
| RewardBench | 12 | Included | ODC-By-1.0 AND MIT (selected components) | All 12 selected entries are mapped to LLMBar, HumanEvalPack, or PRM800K. The original RewardBench component table and publisher notices identify MIT terms for these sources. Retain collection ODC-By attribution and all applicable component notices, including HumanEval and MATH. |
| RULER | 16 | Included | Apache-2.0 | The pinned repository is Apache-2.0. These 16 cases are locally generated retrieval probes inspired by RULER, not unchanged RULER benchmark items; preserve that provenance. |
| SelfAware | 56 | Included | CC-BY-SA-4.0 | The README explicitly distinguishes Apache-2.0 code from CC BY-SA 4.0 DATA. Release these adapted entries under CC BY-SA 4.0 with source credit; do not relabel the data Apache or MIT. |
| Social IQa | 18 | Included | CC-BY-4.0 | The publisher's allenai/social_i_qa dataset card explicitly states CC BY 4.0; retain the Sap et al. citation, original download URL, and normalization notice. |
| SPARTQA | 36 | Included | MIT | The pinned original repository LICENSE.md is MIT. Retain its copyright notice and identify normalized answer choices. |
| SST-2 | 6 | Recover locally | Not established | The publisher dataset card lists the license as unknown, and no redistribution grant for the selected movie-review text was established. Obtain the original files for local recovery. |
| ToMBench | 31 | Included | MIT | The pinned original repository LICENSE is MIT. Retain its notice and the source question identifiers. |
| ToxicChat | 6 | Included | CC-BY-NC-4.0 | Included for the stated noncommercial research release under CC BY-NC 4.0, which the pinned publisher card applies to both user prompts and model outputs. Retain creator credit, the license notice and normalization disclosures. |
| TREC | 6 | Recover locally | Not established | The original question-classification page and downloaded label files provide no redistribution license. Five selected records are from the training collection and one from TREC 10; no blanket government-work assumption is made for the combined collection. |
| When2Call | 57 | Included | CC-BY-4.0 | The pinned NVIDIA dataset card explicitly licenses the dataset under CC BY 4.0. Retain source credit and describe the choice-format normalization. |
| WildGuardTest | 6 | Included | ODC-By-1.0 | The six selected records come from the synthetic WildGuardTest split described in the original paper, section 3.2. Retain the publisher’s ODC-By terms, source credit, original row locators and normalization notices. |
Recover question texts · PIQA recovery and AFL terms
Attribution and adaptation
CLUTRR, EmoWOZ and ToxicChat question material is included for this noncommercial research release under CC BY-NC 4.0. Its noncommercial restriction remains applicable to downstream reuse. The full license and disclaimer are in licenses/CC-BY-NC-4.0.txt.
Question records retain the source URL, revision when available, original locator, attribution, and per-source license. Where the evaluated choice format differs from the source question, released records also include the native representation. Adapted SelfAware entries remain under CC BY-SA 4.0; BoolQ entries retain CC BY-SA 3.0. Per-question Wikipedia, OANC, BIG-bench and component attribution is recorded in data/source_attributions.json and displayed on the question pages. This is a collection of separately licensed source materials; there is no blanket MIT license for the questions.
The original license notices are in licenses/sources/. data/sources.json records checked evidence URLs, source hashes, and license excerpts. data/sharing_decisions.csv gives a decision for each of the 1,005 question IDs.
Publisher dataset-card declarations and source credits are recorded in data/sources.json. Follow each source’s terms when obtaining or reusing its text.