Spatial cognition · question 14
Back to Spatial cognition · All capabilities
Question ID: spartqa/1be52e164cba70ab5683
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
We have two blocks, A and B. Block A contains one medium yellow square. There is also a medium black square in this block. Below the medium yellow square is the medium black square. Block B is to the left of block A. It contains two medium yellow squares. Below medium yellow square number two is medium yellow square number one. A medium black square is touching the bottom edge of this block. It is below medium yellow square number one and medium yellow square number two.
Question
Which object is to the left of a medium black square? medium yellow square number two or medium yellow square number one?
Answer choices
- 0: medium yellow square number two
- 1: medium yellow square number one
- 2: both of them
- 3: none of them
Reference answer: both of them (2)
Exact evaluated input as JSON
{
"state": "We have two blocks, A and B. Block A contains one medium yellow square. There is also a medium black square in this block. Below the medium yellow square is the medium black square. Block B is to the left of block A. It contains two medium yellow squares. Below medium yellow square number two is medium yellow square number one. A medium black square is touching the bottom edge of this block. It is below medium yellow square number one and medium yellow square number two.",
"question": "Which object is to the left of a medium black square? medium yellow square number two or medium yellow square number one?",
"choices": {
"0": "medium yellow square number two",
"1": "medium yellow square number one",
"2": "both of them",
"3": "none of them"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | both of them | Correct |
| Opus 5.5 | both of them | Correct |
| GLiDE v1.2 | both of them | Correct |
| Decider v1.1 | both of them | Correct |
| Jev | both of them | Correct |
| Microsoft Decision-1 | both of them | Correct |
| Clef | both of them | Correct |
| Luna Decisions | both of them | Correct |
| Clef Flash | both of them | Correct |
| Strands Decider | both of them | Correct |
| GLiNER2.5 Decide | medium yellow square number one | Incorrect |
| Laya | both of them | Correct |
| CalDec v1 | both of them | Correct |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
"record": "3587/2",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "4a769938cd98b47af4217c3ea2e23c4a29b942c1b460af2ff7a6a5b272139d9b"
}