RaindropA practical guide to choosing decision modelsGitHub ↗

Source terms

Each dataset’s terms still apply. In the table below, Included means we provide the question text. Recover locally means you download the original files and use our tools to reconstruct the questions we used. We checked the source terms on November 9, 2026.

The collection keeps all 1,005 original-question metadata records and all 13,065 baseline outcomes. Full inputs are bundled for 857 questions; the other 148 can be reconstructed locally from original source files under the source terms. Recovered question texts stay in the ignored .local/ folder and are not redistributed with the repository.

Source Questions Question text Terms Decision and remaining conditions
AG News 6 Recover locally Not established No explicit redistribution grant for the original news excerpts was established. Obtain the selected text from the original source under its applicable terms.
AgentProcessBench 4 Included MIT The dataset page linked by the official repository declares MIT for the dataset. All four selected trajectories are from tau2, whose original repository also provides an MIT license. Retain both publisher attribution and the original Sierra Research notice.
BIG-bench Analogical Similarity 67 Included Apache-2.0 The pinned BIG-bench repository provides Apache-2.0 terms. Retain the repository notice and task-specific provenance.
bAbI 72 Included CC-BY-3.0 The downloaded original archive contains LICENSE.txt naming Facebook and CC BY 3.0. Retain that notice and mark the multiple-choice adaptations.
Banking77 7 Included CC-BY-4.0 The pinned publisher dataset card explicitly declares CC BY 4.0. Retain source attribution and normalization notes.
BIG-Bench Hard selected tasks 89 Included MIT AND Apache-2.0 The pinned BBH repository has an MIT license; it selects tasks from BIG-bench. Retain both the BBH MIT notice and the upstream BIG-bench Apache-2.0 notice, plus task identifiers.
BigToM 17 Included MIT The original pinned repository includes an MIT license and author copyright notice. Retain it for the selected synthetic scenarios.
BoolQ 17 Included CC-BY-SA-3.0 The 17 passage/question pairs include original Wikipedia article and contributor-history links, creator credit, normalization notices and CC BY-SA 3.0 terms. The source files match the original Google release checksums.
CLadder 72 Included MIT The pinned original repository has an MIT license. Retain its notice and source IDs.
CLINC150 7 Included CC-BY-3.0 The pinned source LICENSE is CC BY 3.0. Retain source credit and describe normalization.
CLUTRR 33 Included CC-BY-NC-4.0 Included for the stated noncommercial research release under the original CLUTRR CC BY-NC 4.0 license. Retain creator credit, the license notice and normalization disclosures; the mirror card alone is not the licensing basis.
ContractNLI 16 Included CC-BY-4.0 The original dataset LICENSE and extracted upstream README specify CC BY 4.0. Retain credit, the license, and the original document/question identifiers.
CRASS 10 Included Apache-2.0 The pinned BIG-bench source provides Apache-2.0 terms; retain its notice and the crass_ai task provenance.
CRUXEval 19 Included MIT The pinned README explicitly says CRUXEval is MIT licensed; its LICENSE includes Meta's copyright notice. Retain both source credit and the adaptation notice.
EmoWOZ 6 Included CC-BY-NC-4.0 Included for the stated noncommercial research release under the publisher-declared CC BY-NC 4.0 terms. Retain creator credit, the license notice and normalization disclosures.
FANToM 17 Recover locally MIT (project code) The project code is MIT licensed, but redistribution terms for the separately hosted dataset archive are not explicit. Obtain the 17 dialogues from the publisher and preserve its evaluation-only intended-use statement.
FinQA 10 Included MIT The pinned original repository LICENSE is MIT. Retain its notice and describe the choice-format adaptation; no claim of independent rights over other SEC material is added.
HLE-MultiChoice 7 Included MIT The pinned publisher dataset card declares MIT. Retain publisher attribution and the dataset-card license declaration.
iSarcasmEval 29 Included MIT The pinned original repository provides an MIT license covering its distributed material; retain the original copyright notice. No claim of public-domain status is made for the messages.
List Functions 34 Included Apache-2.0 All 34 selected examples were matched directly to Apache-licensed BIG-bench task files, following the upstream author’s source explanation. The complete seen example pairs and evaluated test input/output values match. Retain the BIG-bench notice and per-question upstream locators.
MiniSCAN 33 Recover locally Not established These questions use a pseudoword adaptation whose exact source files have no established redistribution grant. Obtain those original files for local recovery; another task version or mirror’s license does not establish terms for this version.
MMLU 28 Included MIT The pinned publisher card explicitly declares MIT and links the original benchmark license. Retain attribution and identify normalized multiple-choice labels.
MNLI 6 Included OANC (selected non-fiction genres) The six selected pairs use slate, telephone, oup, verbatim, government and letters, all documented OANC genres. Include the OANC license and Appendix I credits, per-question genre/pair identifiers and source links, and transparent-modification attribution/date. No fiction entries are in this selection.
NEWTON 31 Recover locally Not established No explicit dataset redistribution license was established for the selected source files. Obtain them from the publisher for local recovery; the README’s illustrative LICENSE entry is not a license grant.
PIQA 30 Recover locally AFL-3.0 PIQA uses AFL 3.0, including a recipient-assent condition. The recovery tool requires license acceptance, downloads the original files and verifies the 30 selected inputs in local storage.
ProofWriter 19 Recover locally Not established The original ProofWriter archive and paper provide no dataset license. The separate RuleTaker repository’s Apache license does not explicitly cover this later ProofWriter release; conflicting mirror labels remain insufficient.
PUB 31 Included MIT The pinned publisher dataset card declares MIT. Retain that declaration, the publisher attribution, and task-specific source locators.
QuestBench (Google DeepMind) 38 Included CC-BY-4.0 The original archive LICENSE distinguishes Apache-2.0 software from CC BY 4.0 other materials. The selected questions are dataset entries; retain Google copyright and the archive notice.
RewardBench 12 Included ODC-By-1.0 AND MIT (selected components) All 12 selected entries are mapped to LLMBar, HumanEvalPack, or PRM800K. The original RewardBench component table and publisher notices identify MIT terms for these sources. Retain collection ODC-By attribution and all applicable component notices, including HumanEval and MATH.
RULER 16 Included Apache-2.0 The pinned repository is Apache-2.0. These 16 cases are locally generated retrieval probes inspired by RULER, not unchanged RULER benchmark items; preserve that provenance.
SelfAware 56 Included CC-BY-SA-4.0 The README explicitly distinguishes Apache-2.0 code from CC BY-SA 4.0 DATA. Release these adapted entries under CC BY-SA 4.0 with source credit; do not relabel the data Apache or MIT.
Social IQa 18 Included CC-BY-4.0 The publisher's allenai/social_i_qa dataset card explicitly states CC BY 4.0; retain the Sap et al. citation, original download URL, and normalization notice.
SPARTQA 36 Included MIT The pinned original repository LICENSE.md is MIT. Retain its copyright notice and identify normalized answer choices.
SST-2 6 Recover locally Not established The publisher dataset card lists the license as unknown, and no redistribution grant for the selected movie-review text was established. Obtain the original files for local recovery.
ToMBench 31 Included MIT The pinned original repository LICENSE is MIT. Retain its notice and the source question identifiers.
ToxicChat 6 Included CC-BY-NC-4.0 Included for the stated noncommercial research release under CC BY-NC 4.0, which the pinned publisher card applies to both user prompts and model outputs. Retain creator credit, the license notice and normalization disclosures.
TREC 6 Recover locally Not established The original question-classification page and downloaded label files provide no redistribution license. Five selected records are from the training collection and one from TREC 10; no blanket government-work assumption is made for the combined collection.
When2Call 57 Included CC-BY-4.0 The pinned NVIDIA dataset card explicitly licenses the dataset under CC BY 4.0. Retain source credit and describe the choice-format normalization.
WildGuardTest 6 Included ODC-By-1.0 The six selected records come from the synthetic WildGuardTest split described in the original paper, section 3.2. Retain the publisher’s ODC-By terms, source credit, original row locators and normalization notices.

Recover question texts · PIQA recovery and AFL terms

Attribution and adaptation

CLUTRR, EmoWOZ and ToxicChat question material is included for this noncommercial research release under CC BY-NC 4.0. Its noncommercial restriction remains applicable to downstream reuse. The full license and disclaimer are in licenses/CC-BY-NC-4.0.txt.

Question records retain the source URL, revision when available, original locator, attribution, and per-source license. Where the evaluated choice format differs from the source question, released records also include the native representation. Adapted SelfAware entries remain under CC BY-SA 4.0; BoolQ entries retain CC BY-SA 3.0. Per-question Wikipedia, OANC, BIG-bench and component attribution is recorded in data/source_attributions.json and displayed on the question pages. This is a collection of separately licensed source materials; there is no blanket MIT license for the questions.

The original license notices are in licenses/sources/. data/sources.json records checked evidence URLs, source hashes, and license excerpts. data/sharing_decisions.csv gives a decision for each of the 1,005 question IDs.

Publisher dataset-card declarations and source credits are recorded in data/sources.json. Follow each source’s terms when obtaining or reusing its text.

Type to search questions, variants and documentation.