Spatial cognition · question 12
Back to Spatial cognition · All capabilities
Question ID: spartqa/6cea3e0444d515c7f763
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
There are two blocks, A and B. Block A is below B. Block A has a medium black diamond. Block B contains one big yellow oval and a little black diamond. There is also one big blue oval touching the bottom edge of this block. A big blue rectangle is to the right side of, close to and on the top of the big blue oval. Below and to the left of the little black diamond there is the big yellow oval. The little black diamond is touching the right side edge of this block.
Question
What is the relation between the medium shape and the big blue shape that is on the top of a big blue oval which is touching the bottom edge of a block?
Answer choices
- 0: left
- 1: right
- 2: top
- 3: below
- 4: close to
- 5: far from
- 6: touching
- 7: DK
Reference answer: below (3)
Exact evaluated input as JSON
{
"state": "There are two blocks, A and B. Block A is below B. Block A has a medium black diamond. Block B contains one big yellow oval and a little black diamond. There is also one big blue oval touching the bottom edge of this block. A big blue rectangle is to the right side of, close to and on the top of the big blue oval. Below and to the left of the little black diamond there is the big yellow oval. The little black diamond is touching the right side edge of this block.",
"question": "What is the relation between the medium shape and the big blue shape that is on the top of a big blue oval which is touching the bottom edge of a block?",
"choices": {
"0": "left",
"1": "right",
"2": "top",
"3": "below",
"4": "close to",
"5": "far from",
"6": "touching",
"7": "DK"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | below | Correct |
| Opus 5.5 | below | Correct |
| GLiDE v1.2 | below | Correct |
| Decider v1.1 | below | Correct |
| Jev | left | Incorrect |
| Microsoft Decision-1 | below | Correct |
| Clef | far from | Incorrect |
| Luna Decisions | left | Incorrect |
| Clef Flash | far from | Incorrect |
| Strands Decider | below | Correct |
| GLiNER2.5 Decide | touching | Incorrect |
| Laya | below | Correct |
| CalDec v1 | below | Correct |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/unseen_test.json",
"record": "268/5",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "3ae84ccc847ba883a494a4a4b7caa7665b4b758f4107432e0666b208cee56d10"
}