Spatial cognition · question 64
Back to Spatial cognition · All capabilities
Question ID: spartqa/fc7257c11e5c71fcb907
Source: SPARTQA
Input format: adapted choices
Explore this question’s altered versions
Context
We have three blocks, A, B and C. Block B is below blocks C and A. Block A contains one medium blue square. The medium blue square is below a medium yellow square. Block B has two medium black squares. Below medium black square number two there is medium black square number one. Block C contains a medium black square.
Question
Which blocks don't have any medium shape that is below a black square?
Answer choices
- A: ["A"]
- B: []
- C: ["B"]
- D: ["B", "C"]
- E: ["A", "B", "C"]
- F: ["A", "C"]
- G: ["A", "B"]
- H: ["C"]
Reference answer: ["A", "C"] (F)
Exact evaluated input as JSON
{
"state": "We have three blocks, A, B and C. Block B is below blocks C and A. Block A contains one medium blue square. The medium blue square is below a medium yellow square. Block B has two medium black squares. Below medium black square number two there is medium black square number one. Block C contains a medium black square.",
"question": "Which blocks don't have any medium shape that is below a black square?",
"choices": {
"A": "[\"A\"]",
"B": "[]",
"C": "[\"B\"]",
"D": "[\"B\", \"C\"]",
"E": "[\"A\", \"B\", \"C\"]",
"F": "[\"A\", \"C\"]",
"G": "[\"A\", \"B\"]",
"H": "[\"C\"]"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | ["A", "C"] | Correct |
| Opus 5.5 | ["A", "C"] | Correct |
| GLiDE v1.2 | ["A", "C"] | Correct |
| Decider v1.1 | ["A", "C"] | Correct |
| Jev | ["A", "C"] | Correct |
| Microsoft Decision-1 | ["A", "C"] | Correct |
| Clef | ["A", "C"] | Correct |
| Luna Decisions | ["A", "C"] | Correct |
| Clef Flash | ["A", "C"] | Correct |
| Strands Decider | ["A"] | Incorrect |
| GLiNER2.5 Decide | ["A"] | Incorrect |
| Laya | ["B", "C"] | Incorrect |
| CalDec v1 | ["B"] | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
"record": "10509/6",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "a8b17309e0b3948b8d3d261961166140ab0c576218e4e346b3675c62f670924a"
}