Similar distractors · Add fewer similar but irrelevant items variant
Original question · Other versions of this question · All variants
This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.
Changed fields: question instructions.
How the edits were constructed
Context
We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.
Question
Which blocks don't have any medium square that is below a medium square?
«Is the teal cylinder Ovelin above the amber prism Nerath?»
Answer choices
- A. ["A", "C"]
- B. ["A", "B", "C"]
- C. ["B", "C"]
- D. ["A"]
- E. ["C"]
- F. ["B"]
- G. ["A", "B"]
- H. []
Reference answer: E.
Exact evaluated input as JSON
{
"state": "We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.",
"question": "Which blocks don't have any medium square that is below a medium square?\n\n«Is the teal cylinder Ovelin above the amber prism Nerath?»",
"choices": {
"A": "[\"A\", \"C\"]",
"B": "[\"A\", \"B\", \"C\"]",
"C": "[\"B\", \"C\"]",
"D": "[\"A\"]",
"E": "[\"C\"]",
"F": "[\"B\"]",
"G": "[\"A\", \"B\"]",
"H": "[]"
}
}
Recorded model answers
The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.
| Model | Original answer | Variant answer | Change |
|---|---|---|---|
| GLiDE v1.2 | E — ["C"] |
E — ["C"] |
Same answer |
| Decider v1.1 | E — ["C"] |
E — ["C"] |
Same answer |
| Jev | E — ["C"] |
E — ["C"] |
Same answer |
| Microsoft Decision-1 | E — ["C"] |
E — ["C"] |
Same answer |
| Clef | E — ["C"] |
E — ["C"] |
Same answer |
| Luna Decisions | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
| Clef Flash | E — ["C"] |
E — ["C"] |
Same answer |
| Strands Decider | A — ["A", "C"] |
H — [] |
Different answer |
| GLiNER2.5 Decide | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
| Laya | B — ["A", "B", "C"] |
A — ["A", "C"] |
Different answer |
| CalDec v1 | A — ["A", "C"] |
A — ["A", "C"] |
Same answer |
Identifiers and recorded conditions
Variant ID: spartqa/12df91f6e1630eeec74a/similar_distractors/low/clean-source_first_spartqa_near_questions_full_0223
Input fingerprint: 231b0856eaace08efac233545d1b142c6e3d3eaa7e502735619d6d58c2ad8987
| Change | Recorded version |
|---|---|
| Similar distractors | Add fewer similar but irrelevant items |
Source attribution and dataset terms are linked from the original question.