Spatial cognition · question 2
Back to Spatial cognition · All capabilities
Question ID: spartqa/c941ed87aa194a1bbe21
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
There are three blocks, A, B and C. Block A is to the left of C and block B is below A. Block A contains two small blue squares. There is also a medium blue circle in this block. To the right of small blue square number two there is the medium blue circle. A medium yellow square is above a big yellow triangle. The big yellow triangle is to the right of small blue square number two. Small blue square number one is touching the right edge of this block. The medium yellow square is near to and above small blue square number one. Block B contains a big yellow circle. The big yellow circle is to the right of a big yellow triangle. And block C has a small blue triangle and a medium blue circle. The small blue triangle is touching the right edge of this block. The medium blue circle is touching the bottom edge of this block. Above and to the right of the medium blue circle there is the small blue triangle.
Question
Are all medium blue circles that are to the left of a small blue triangle that is touching the right edge of a block , near to all yellow objects that are below a medium circle that is in block A?
Answer choices
- Yes: Yes
- No: No
- DK: DK
Reference answer: DK (DK)
Exact evaluated input as JSON
{
"state": "There are three blocks, A, B and C. Block A is to the left of C and block B is below A. Block A contains two small blue squares. There is also a medium blue circle in this block. To the right of small blue square number two there is the medium blue circle. A medium yellow square is above a big yellow triangle. The big yellow triangle is to the right of small blue square number two. Small blue square number one is touching the right edge of this block. The medium yellow square is near to and above small blue square number one. Block B contains a big yellow circle. The big yellow circle is to the right of a big yellow triangle. And block C has a small blue triangle and a medium blue circle. The small blue triangle is touching the right edge of this block. The medium blue circle is touching the bottom edge of this block. Above and to the right of the medium blue circle there is the small blue triangle.",
"question": "Are all medium blue circles that are to the left of a small blue triangle that is touching the right edge of a block , near to all yellow objects that are below a medium circle that is in block A?",
"choices": {
"Yes": "Yes",
"No": "No",
"DK": "DK"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | DK | Correct |
| Opus 5.5 | DK | Correct |
| GLiDE v1.2 | DK | Correct |
| Decider v1.1 | DK | Correct |
| Jev | No | Incorrect |
| Microsoft Decision-1 | No | Incorrect |
| Clef | Yes | Incorrect |
| Luna Decisions | Yes | Incorrect |
| Clef Flash | No | Incorrect |
| Strands Decider | No | Incorrect |
| GLiNER2.5 Decide | Yes | Incorrect |
| Laya | Yes | Incorrect |
| CalDec v1 | Yes | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
"record": "7521/1",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "aecd42d841fb9f2095051d5bdaa62f3a9f28e9cf78baba83ee0fc49a2963ea41"
}