Spatial cognition · question 58
Back to Spatial cognition · All capabilities
Question ID: spartqa/356448f6aa44b12229af
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
We have three blocks, A, B and C. Block A has one big black triangle. The big black triangle is touching the right edge of this block. This block also has a big yellow circle. Near to, to the right of, touching and above the big yellow circle there is the big black triangle. The big yellow circle is touching the bottom edge of this block. Then, we have block B. It has a medium black circle and a medium yellow square. The medium yellow square is to the right of and below a medium blue circle. The medium black circle is near to and above the medium blue circle. To the left of block A and above block B there is block C with a medium blue square and a big yellow square in it. Below the big yellow square there is the medium blue square.
Question
Is there a square that is below a blue square , to the right of a medium blue thing that is near to a black circle?
Answer choices
- Yes: Yes
- No: No
- DK: DK
Reference answer: Yes (Yes)
Exact evaluated input as JSON
{
"state": "We have three blocks, A, B and C. Block A has one big black triangle. The big black triangle is touching the right edge of this block. This block also has a big yellow circle. Near to, to the right of, touching and above the big yellow circle there is the big black triangle. The big yellow circle is touching the bottom edge of this block. Then, we have block B. It has a medium black circle and a medium yellow square. The medium yellow square is to the right of and below a medium blue circle. The medium black circle is near to and above the medium blue circle. To the left of block A and above block B there is block C with a medium blue square and a big yellow square in it. Below the big yellow square there is the medium blue square.",
"question": "Is there a square that is below a blue square , to the right of a medium blue thing that is near to a black circle?",
"choices": {
"Yes": "Yes",
"No": "No",
"DK": "DK"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Yes | Correct |
| Opus 5.5 | Yes | Correct |
| GLiDE v1.2 | Yes | Correct |
| Decider v1.1 | Yes | Correct |
| Jev | No | Incorrect |
| Microsoft Decision-1 | No | Incorrect |
| Clef | No | Incorrect |
| Luna Decisions | No | Incorrect |
| Clef Flash | No | Incorrect |
| Strands Decider | Yes | Correct |
| GLiNER2.5 Decide | Yes | Correct |
| Laya | Yes | Correct |
| CalDec v1 | Yes | Correct |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
"record": "6316/1",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "6acfeeeaee39cfff87e65d9346bb566f657ccdf887e65bb566b2cf0e30e552eb"
}