RaindropA practical guide to choosing decision modelsGitHub ↗

Variants of mmlu/372a72b278f6e7e4df70

Original question · Capability index · Variant methods

Each row is a distinct evaluated input. An input can serve more than one recorded condition; those aliases are listed on its page.

Variant Text Correct / answered
Extra events · Add 12 unrelated events Included 4 / 11
Extra events · Add 2 unrelated events Included 3 / 11
Extra events · Add 6 unrelated events Included 4 / 11
Evidence position · Place the relevant information at the end Included 4 / 11
Evidence position · Place the relevant information in the middle Included 3 / 11
Evidence position · Place the relevant information at the beginning Included 4 / 11
Repetition · Present the same evidence twice Included 4 / 11
Repetition · Present the same evidence three times Included 2 / 11
Unrelated examples · No worked example (absence-control recipe) Included 4 / 11
Unrelated examples · Add examples unrelated to the question Included 7 / 11
Irrelevant details · Add a larger amount of irrelevant detail Included 4 / 11
Irrelevant details · Add a small amount of irrelevant detail Included 4 / 11
Misleading details · Add an unverified claim that agrees with the evidence Included 10 / 11
Misleading details · Add an unverified claim that conflicts with the evidence Included 2 / 11
Option order · Reverse the answer choices Included 3 / 11
Option order · Move the first answer choice to the end Included 6 / 11
Paraphrasing · Change the sentence structure Included 4 / 11
Paraphrasing · Change individual words and phrases Included 4 / 11
Evidence order · Present the evidence in reverse order Included 4 / 11
Similar distractors · Add more similar but irrelevant items Included 3 / 11
Similar distractors · Add fewer similar but irrelevant items Included 4 / 11
Text length · 2,048-token input target Included 6 / 11
Text length · 512-token input target Included 5 / 11
Text length · 8,192-token input target Included 3 / 7

Type to search questions, variants and documentation.