RaindropA practical guide to choosing decision modelsGitHub ↗

How variants were made

Browse variants · Results · Verify or rerun

The context-sensitivity study uses 24,563 distinct altered inputs derived from the same 1,005 original questions, across 12 kinds of change. Each original has multiple versions; the number varies because not every edit applies to every question. Eleven decision models were tested. Astra and Opus are baseline comparators only.

The variants change the context, question wording or instructions, or answer-choice order while retaining the recorded reference answer. The exact evaluated text is saved: reproducing a variant does not require asking another language model to rewrite it.

The changes

Change What the versions vary
Text length Added text targeting 512, 2,048 or 8,192 tokens, with additional source-specific length versions. These are input targets, rather than a guarantee of the same token count under every model's tokenizer.
Extra events Two, six or twelve unrelated events alongside the original information.
Evidence position The same decisive information near the beginning, middle or end of a surrounding passage.
Evidence order Reversed or shuffled presentation of the same information. Local rearrangements and edits that also change wording have separate records.
Repetition The same evidence presented twice or three times.
Invented names Replacement familiar names compared with invented names.
Irrelevant details Smaller or larger amounts of information unrelated to the answer.
Similar distractors Fewer or more irrelevant items resembling the information needed to answer.
Misleading details An added reader assessment that suggests either the correct answer or an incorrect answer. These are claims about the answer, rather than changes to the underlying evidence.
Unrelated examples Worked examples unrelated to the target judgment, with separately constructed versions without the examples or with prose in their place.
Paraphrasing Changes to words and phrases, or to sentence structure.
Option order Reversing the choices or moving the first choice to the end.

Edits are adapted to the original source format. An added passage can appear in the context or in the question instructions. Some versions use shared construction recipes across a source family; others have question-specific edits. The individual pages show the changed fields, complete text where shareable, recorded condition names and input fingerprint.

A reference answer is the answer key used for scoring, not a model's prediction. Retaining the key records the intended answer-preserving design. It does not establish that every authored rewrite is semantically equivalent; inspect an individual input before drawing conclusions from an unexpected answer.

Which versions are compared?

An edit's stored label alone is not enough to establish a controlled comparison. For example, a version labelled “reversed” may reverse the complete evidence, move a single phrase, or combine rearrangement with a wording change. The reported ordering comparison uses only the exact questions identified as complete-information permutations.

The explicit lists in comparisons.json identify the versions of each original used together. Each family records its original question, the input ID at each level and construction information. These lists were established from the inputs and construction records, rather than from which models answered correctly.

Comparison Original questions with all required inputs fitting every model
512- vs 2,048-token targets 733
2, 6 and 12 extra events 1,005
Beginning, middle and end 944
Original, reversed and shuffled evidence 335
Evidence repeated twice vs three times 989
Familiar vs invented names 103
Low vs high irrelevant detail 1,004
Low vs high similar distractors 988
Accurate vs misleading reader assessment 1,004
Worked examples vs their authored no-example version 592
Worked examples vs a prose control 241
Worked examples vs a natural-prose control 71
Word substitutions vs sentence restructuring 989
Original, rotated and reversed answer choices 673

Unrelated examples are compared with versions that omit the examples or replace them with ordinary prose. These are separate comparisons, and the versions used for comparison can differ from the unedited original. Additional variants outside the comparisons above remain available on the individual pages.

Linking each variant to its original answer

Some original responses were reused from earlier runs with matching inputs and model settings. Unchanged questions were not rerun for every model, so individual answer changes may also reflect variation between runs.

Every variant outcome has an original_result_id pointing to the exact response in original-controls.jsonl. The individual variant pages display that response next to the variant answer. Proven identical native model inputs can also share a result; input_alias, source_case_id and source_input_hash preserve that relationship.

Scoring and coverage

Each comparison shows accuracy and answer changes, including mistakes introduced or corrected. Overall accuracy can stay the same even when individual answers change. Scores use questions with valid responses to every version required for the comparison; counts are shown for each model. Multiple versions of one question are not independent questions.

Clef and Clef Flash sort answer choices before answering. Their option-order results reflect this preprocessing.

Sharing the text

Full evaluated text is included for 20,961 variants. The other 3,602 derive from originals whose terms do not clearly support republishing their text here, or impose additional requirements. Download those originals under their source terms and recover the selected variants locally. Saved edit recipes combine authored additions with references into your original files, and verify the reconstructed input fingerprint. All IDs, reference keys, model outcomes and comparison lists are included regardless of text availability.

Type to search questions, variants and documentation.