Irrelevant details · Add a small amount of irrelevant detail variant
Original question · Other versions of this question · All variants
This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.
Changed fields: context.
How the edits were constructed
Context
The notebook has a linen cover.
We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.
Question
Which blocks don't have any medium square that is below a medium square?
Answer choices
- A. ["A", "C"]
- B. ["A", "B", "C"]
- C. ["B", "C"]
- D. ["A"]
- E. ["C"]
- F. ["B"]
- G. ["A", "B"]
- H. []
Reference answer: E.
Exact evaluated input as JSON
{
"state": "The notebook has a linen cover.\n\nWe have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.",
"question": "Which blocks don't have any medium square that is below a medium square?",
"choices": {
"A": "[\"A\", \"C\"]",
"B": "[\"A\", \"B\", \"C\"]",
"C": "[\"B\", \"C\"]",
"D": "[\"A\"]",
"E": "[\"C\"]",
"F": "[\"B\"]",
"G": "[\"A\", \"B\"]",
"H": "[]"
}
}
Recorded model answers
The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.
| Model | Original answer | Variant answer | Change |
|---|---|---|---|
| GLiDE v1.2 | E — ["C"] |
E — ["C"] |
Same answer |
| Decider v1.1 | E — ["C"] |
E — ["C"] |
Same answer |
| Jev | E — ["C"] |
E — ["C"] |
Same answer |
| Microsoft Decision-1 | E — ["C"] |
E — ["C"] |
Same answer |
| Clef | E — ["C"] |
E — ["C"] |
Same answer |
| Luna Decisions | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
| Clef Flash | E — ["C"] |
E — ["C"] |
Same answer |
| Strands Decider | A — ["A", "C"] |
H — [] |
Different answer |
| GLiNER2.5 Decide | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
| Laya | B — ["A", "B", "C"] |
A — ["A", "C"] |
Different answer |
| CalDec v1 | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
Identifiers and recorded conditions
Variant ID: spartqa/12df91f6e1630eeec74a/irrelevant_details/low/clean-source_first_spartqa_short_contexts_full_0136
Input fingerprint: 372edd707181222bcccd5c3d1825a22d5b800627588e52fc3c6f8ef0d97952ba
| Change | Recorded version |
|---|---|
| Irrelevant details | Add a small amount of irrelevant detail |
Source attribution and dataset terms are linked from the original question.