Spatial cognition · question 54
Back to Spatial cognition · All capabilities
Question ID: spartqa/12df91f6e1630eeec74a
Source: SPARTQA
Input format: adapted choices
Explore this question’s altered versions
Context
We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.
Question
Which blocks don't have any medium square that is below a medium square?
Answer choices
- A: ["A", "C"]
- B: ["A", "B", "C"]
- C: ["B", "C"]
- D: ["A"]
- E: ["C"]
- F: ["B"]
- G: ["A", "B"]
- H: []
Reference answer: ["C"] (E)
Exact evaluated input as JSON
{
"state": "We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.",
"question": "Which blocks don't have any medium square that is below a medium square?",
"choices": {
"A": "[\"A\", \"C\"]",
"B": "[\"A\", \"B\", \"C\"]",
"C": "[\"B\", \"C\"]",
"D": "[\"A\"]",
"E": "[\"C\"]",
"F": "[\"B\"]",
"G": "[\"A\", \"B\"]",
"H": "[]"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | ["C"] | Correct |
| Opus 5.5 | ["C"] | Correct |
| GLiDE v1.2 | ["C"] | Correct |
| Decider v1.1 | ["C"] | Correct |
| Jev | ["C"] | Correct |
| Microsoft Decision-1 | ["C"] | Correct |
| Clef | ["C"] | Correct |
| Luna Decisions | ["A", "C"] | Incorrect |
| Clef Flash | ["C"] | Correct |
| Strands Decider | ["A", "C"] | Incorrect |
| GLiNER2.5 Decide | ["A", "C"] | Incorrect |
| Laya | ["A", "B", "C"] | Incorrect |
| CalDec v1 | ["A", "C"] | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/dev.json",
"record": "29/6",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "f95adad07f626deff421237e34a1bcaa330d12d84f9eef517aff1da3e8ccdca3"
}