RaindropA practical guide to choosing decision modelsGitHub ↗

Overview

Questions, model answers, and scoring code accompanying Raindrop’s research post. The study compares judgment accuracy across 15 capabilities and examines how changes in wording, framing and context affect those judgments. Inspect individual answers and reproduce the reported scores.

Read the post · Browse questions · Variants · Results · Methods · Reproduce

Original questions Capability areas Baseline models Baseline outcomes
1,005 15, with 67 questions each 11 decision models + 2 LLM baselines 13,065

Start exploring

Context-sensitivity study

The eleven decision models also answered altered versions of these originals: 24,563 variants, of which 23,374 fit every model. Explore every variant, read how the edits were made, or inspect the comparison results. The comparison tables include changed answers, correct-to-wrong and wrong-to-correct judgments, as well as accuracy.

Check the saved variant results offline:

python3 variants.py verify
python3 variants.py summarize

For fresh predictions, use the variant reproduction guide.

Baseline results at a glance

Model Correct / answered Accuracy
Astra (LLM baseline) 999 / 1,005 99.4%
Opus 5.5 (LLM baseline) 997 / 1,003 99.4%
GLiDE v1.2 954 / 1,005 94.9%
Decider v1.1 919 / 1,005 91.4%
Jev 892 / 1,005 88.8%
Microsoft Decision-1 846 / 1,004 84.3%
Clef 835 / 1,005 83.1%
Luna Decisions 777 / 997 77.9%
Clef Flash 768 / 1,005 76.4%
Strands Decider 678 / 1,005 67.5%
GLiNER2.5 Decide 429 / 1,005 42.7%
Laya 416 / 1,005 41.4%
CalDec v1 408 / 1,005 40.6%

Accuracy is calculated over answered questions. Errors and refusals remain visible in coverage. See the capability breakdown and binary error analysis, or inspect every model's answer to a question.

Verify the baseline scores

Download or clone this repository, then open a terminal in the folder containing baseline.py. Python 3.10 or later is enough. No extra packages, credentials, model downloads or paid calls are needed:

python3 baseline.py verify
python3 baseline.py summarize
python3 -m unittest discover -s tests -v

summarize writes fresh CSVs to .local/summaries/, leaving the published tables in summaries/ unchanged. These cover overall accuracy, accuracy by capability, binary error counts and saved cost/timing summaries. verify checks the calculations against the reference tables, as well as input fingerprints, coverage and file checksums.

Get the questions used in the experiment

This repository includes 857 complete questions. The source datasets have different licenses and redistribution terms, so we only bundle question text where those terms clearly support sharing it here. For the remaining 148 questions, permission to redistribute the text is unclear or involves additional requirements.

You can obtain the original files from their publishers under the applicable terms and use the recovery tools below to reconstruct the exact questions we evaluated. Those recovered texts stay in an ignored local folder and are not added to Git.

Question IDs, reference choice keys and saved model outcomes are included for all 1,005 questions. You can verify the reported scores without downloading any question text.

1. Download the original files you need

Choose the datasets you want to reproduce. Follow the publisher's access and use terms, then download the linked original files. You only need sources for the questions you want to recover; the 857 bundled inputs are already in data/questions.jsonl.

Save the files beneath a local source root such as .local/original-sources/, using the relative folder and file names shown below. These links identify the original source versions used for the experiment.

Original dataset Questions used Original files to download; save under the source root
AG News 6 ag_news/data/train-00000-of-00001.parquet
FANToM 17 fantom/fantom.tar.gz
MiniSCAN 33 miniscan/data/scan_rand.jsonl
miniscan/data/scan.jsonl
NEWTON 31 newton/confident_questions.csv
newton/explicit_questions.csv
ProofWriter 19 proofwriter/proofwriter.zip
SST-2 6 sst2/data/train-00000-of-00001.parquet
TREC 6 trec/train_5500.label
trec/test_10.label
PIQA 30 The explicit license-acceptance command below downloads the required original files and recovers the selected questions.

A source file may contain many more questions than we used. The recovery tool extracts only our selected questions. Keep FANToM and ProofWriter as their original archives. For TREC, save the publisher's TREC_10.label as trec/test_10.label. All dataset sources and attribution are listed separately.

For example, download MiniSCAN's two original files into the expected folders:

mkdir -p .local/original-sources/miniscan/data
curl --fail --location \
  --output .local/original-sources/miniscan/data/scan.jsonl \
  "https://raw.githubusercontent.com/linlu-qiu/lm-inductive-reasoning/0eae21f3d3014b92be34c80b23c9120a7b15a51f/data/scan.jsonl"
curl --fail --location \
  --output .local/original-sources/miniscan/data/scan_rand.jsonl \
  "https://raw.githubusercontent.com/linlu-qiu/lm-inductive-reasoning/0eae21f3d3014b92be34c80b23c9120a7b15a51f/data/scan_rand.jsonl"

You can also save the linked files through your browser. Preserve their bytes and names. The tool checks the recorded SHA-256 hashes before reading them; changed files are rejected.

2. Recover just our selected questions

After downloading MiniSCAN:

python3 -B tools/recover_withheld.py \
  --source-root .local/original-sources \
  --datasets miniscan

This writes only the 33 MiniSCAN questions used in the experiment to .local/withheld.questions.jsonl, including the exact evaluated context, question, ordered answer choices and reference answer. The tool checks every reconstructed input against the experiment's saved hash. It reads your downloaded files; it does not download the other datasets or run any models.

To recover multiple downloaded sources together, pass their aliases, for example --datasets miniscan newton. AG News and SST-2 additionally require the optional Parquet reader. The other sources use the Python standard library.

For original URLs, expected file sizes and checksums for a selected source:

python3 -B tools/recover_withheld.py --datasets miniscan --list-files

Recovered text stays in .local/, which Git ignores. Use --output .local/miniscan.questions.jsonl to keep a separately named recovery file. See the full recovery guide for combining multiple files or working with only the questions you have available.

3. Recover PIQA

Read PIQA's AFL 3.0 terms. If you accept them, this command downloads the required original source files and recovers the 30 PIQA questions we used:

python3 -B tools/recover_piqa.py --accept-afl-3.0

The output is .local/piqa.questions.jsonl. PIQA question text is not bundled in this repository.

4. Optionally assemble all 1,005 inputs

Once you have obtained the files for all seven other sources, recover them together by omitting --datasets:

python3 -B tools/recover_withheld.py --source-root .local/original-sources
python3 -B tools/assemble_inputs.py

After PIQA recovery, assembly combines the 857 bundled + 118 recovered + 30 PIQA inputs into .local/questions.complete.jsonl. It checks all 1,005 IDs, complete inputs, choice ordering and reference answers.

To work with the bundled questions and any recovery files you have already created, use python3 tools/assemble_inputs.py --partial. This writes .local/questions.available.jsonl and reports its actual question count; it does not label a partial set as the full baseline.

Inspect one question

The question pages are intended for reading in GitHub. For the machine-readable version:

python3 baseline.py inspect --case bbh/639b88b1fe80634a7d66

To inspect a question recovered from another source, add --local-questions .local/questions.complete.jsonl (or the specific recovery file containing it). The command checks its input fingerprint before displaying the text and recorded answers.

Released inputs contain the complete evaluated state, question, and ordered choices, rather than a shortened display example. They also retain the reference answer and original-source attribution. For questions obtained separately, the recovery workflow supplies the exact input locally.

Run models again

Recover the full 1,005-question set from independently obtained original files. The recovery tools verify the exact inputs and keep held text in ignored local storage.

The reproduction instructions walk through API key setup, sending requests, building the GPU containers, downloading the pinned weights, running all models and scoring fresh answers. Fresh inference requires model access and infrastructure, and can incur API or GPU charges.

Fresh predictions can differ from the saved results as models, routing or sampling change.

You can also connect directly to a provider or test your own model using the same questions and scoring code. The guide explains endpoints, API keys and model IDs, with example connection files.

Repository map

Folder What you will find
questions/ Readable capability indexes and individual question pages
data/ All 1,005 question metadata records, released full inputs, and sharing decisions
results/ All 13,065 baseline outcomes, including failed and refused requests
summaries/ Small CSV tables for checking the reported numbers
evaluation/ Offline request construction and native inference reference code
configs/ Model IDs, pinned checkpoints, runtime settings, and input encoding fingerprints
docs/ Methods, reproduction instructions, and source-sharing notes
licenses/ Original source notices and per-source terms

release_status.json records question coverage and source-sharing counts. provenance.json records input and evaluation provenance. MANIFEST.json contains file checksums.

Attribution and licensing

The questions retain their original dataset licenses and attribution. This repository bundles 857 complete inputs; the other 148 are recovered from original source files. See source terms and notices and question-sharing decisions.

Third-party datasets, downloaded model code and weights, and Raindrop brand artwork retain their own terms.

About this research

Research by Afika Nyati at Raindrop, accompanying A practical guide to choosing decision models (2026). The post explains the findings; the methods and saved question-level outcomes document what was measured.

Data format and scoring

Match a result’s case_id to the question’s id. Match its model to a key in configs/models.json, and its configuration_id to a key in configs/result_configurations.json. JSONL files contain one JSON object per line. The question-file guide and result-field guide describe the fields, choice order, reference answers and input fingerprints.

The scoring definitions explain accuracy, coverage, precision and recall. Use the saved model configurations and measurement conditions when interpreting or reproducing a result.

Citation

Afika Nyati. A practical guide to choosing decision models. Raindrop, 2026. Read the post.

@misc{nyati2026decisionmodels,
  author = {Afika Nyati},
  title = {A practical guide to choosing decision models},
  year = {2026},
  howpublished = {Raindrop},
  url = {https://www.raindrop.ai/blog/choosing-decision-models/}
}

Machine-readable citation metadata is in CITATION.cff.

Questions and corrections

Contact Afika Nyati at afika@raindrop.ai.

For a result or dataset issue, include the question ID, model name, the recorded answer and expected behavior, and enough detail to reproduce the issue. CONTRIBUTING.md explains how to prepare a useful report or a change using the saved data.

Type to search questions, variants and documentation.