Spatial cognition · question 8
Back to Spatial cognition · All capabilities
Question ID: spartqa/707747556e215d5f2917
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
There are three blocks, A, B and C. Block C is to the right of and B is to the left of A. Block A has a medium black circle. Block B has a medium blue square. Block C has one medium blue circle, one small black triangle and a small yellow triangle. The medium blue circle is touching the right edge of this block. To the left of and below the small yellow triangle is the small black triangle. The medium blue circle is to the right of the small yellow triangle. It is above the small black triangle.
Question
Which object is to the right of a square? the small yellow triangle or the small black triangle?
Answer choices
- 0: the small yellow triangle
- 1: the small black triangle
- 2: both of them
- 3: none of them
Reference answer: both of them (2)
Exact evaluated input as JSON
{
"state": "There are three blocks, A, B and C. Block C is to the right of and B is to the left of A. Block A has a medium black circle. Block B has a medium blue square. Block C has one medium blue circle, one small black triangle and a small yellow triangle. The medium blue circle is touching the right edge of this block. To the left of and below the small yellow triangle is the small black triangle. The medium blue circle is to the right of the small yellow triangle. It is above the small black triangle.",
"question": "Which object is to the right of a square? the small yellow triangle or the small black triangle?",
"choices": {
"0": "the small yellow triangle ",
"1": "the small black triangle ",
"2": "both of them",
"3": "none of them"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | both of them | Correct |
| Opus 5.5 | both of them | Correct |
| GLiDE v1.2 | both of them | Correct |
| Decider v1.1 | both of them | Correct |
| Jev | both of them | Correct |
| Microsoft Decision-1 | both of them | Correct |
| Clef | both of them | Correct |
| Luna Decisions | the small yellow triangle | Incorrect |
| Clef Flash | both of them | Correct |
| Strands Decider | both of them | Correct |
| GLiNER2.5 Decide | the small black triangle | Incorrect |
| Laya | both of them | Correct |
| CalDec v1 | the small black triangle | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/dev.json",
"record": "1069/3",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "83e68120b23985a0b47683bc71a7fe67ff263bf9128a57aabb6bd2638473c117"
}