Spatial cognition · question 6
Back to Spatial cognition · All capabilities
Question ID: spartqa/2ecb425c40b3c7e9af6a
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
There are three blocks, A, B and C. Block A contains a large blue rectangle and one small white diamond. The large blue rectangle is to the right side of the small white diamond. Then, we have block B with a midsize blue diamond and a large white circle in it. The large white circle is touching the bottom edge of this block. The midsize blue diamond is touching the bottom edge of this block. It is to the right side of the large white circle. Block C is to the left side of block A and on the top of block B. It contains one small blue circle. To the left side of and under the small blue circle there is a midsize red rectangle.
Question
What is the relation between the large blue rectangle and the small shape that is to the right side of a blue circle?
Answer choices
- 0: left
- 1: right
- 2: top
- 3: under
- 4: close to
- 5: far from
- 6: touching
- 7: DK
Reference answer: right (1)
Exact evaluated input as JSON
{
"state": "There are three blocks, A, B and C. Block A contains a large blue rectangle and one small white diamond. The large blue rectangle is to the right side of the small white diamond. Then, we have block B with a midsize blue diamond and a large white circle in it. The large white circle is touching the bottom edge of this block. The midsize blue diamond is touching the bottom edge of this block. It is to the right side of the large white circle. Block C is to the left side of block A and on the top of block B. It contains one small blue circle. To the left side of and under the small blue circle there is a midsize red rectangle.",
"question": "What is the relation between the large blue rectangle and the small shape that is to the right side of a blue circle?",
"choices": {
"0": "left",
"1": "right",
"2": "top",
"3": "under",
"4": "close to",
"5": "far from",
"6": "touching",
"7": "DK"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | right | Correct |
| Opus 5.5 | right | Correct |
| GLiDE v1.2 | right | Correct |
| Decider v1.1 | right | Correct |
| Jev | right | Correct |
| Microsoft Decision-1 | right | Correct |
| Clef | far from | Incorrect |
| Luna Decisions | DK | Incorrect |
| Clef Flash | far from | Incorrect |
| Strands Decider | right | Correct |
| GLiNER2.5 Decide | right | Correct |
| Laya | under | Incorrect |
| CalDec v1 | under | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/unseen_test.json",
"record": "848/5",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "29ab9b6a809411d7d9a81c649f131e77c7d6a455ed8540ea18c3c9fe72f167a2"
}