Spatial cognition · question 50
Back to Spatial cognition · All capabilities
Question ID: spartqa/5c9863f43a411486d1b2
Source: SPARTQA
Input format: native choices
Explore this question’s altered versions
Context
There are three blocks. Lets call them A, B and C. Block B is to the right of A and block C is above B. Block A has one medium blue triangle and a small yellow circle. To the right of and above the small yellow circle there is the medium blue triangle. Block B has a big blue square, one medium black triangle and one medium yellow triangle. The medium yellow triangle is to the right of and above a small blue triangle. The big blue square is to the right of and above the medium yellow triangle. The small blue triangle is touching the bottom edge of this block.. The medium black triangle is near to and above the small blue triangle. And block C has one medium blue square.
Question
Is the medium blue square , below the small yellow object that is to the left of a black triangle?
Answer choices
- Yes: Yes
- No: No
- DK: DK
Reference answer: DK (DK)
Exact evaluated input as JSON
{
"state": "There are three blocks. Lets call them A, B and C. Block B is to the right of A and block C is above B. Block A has one medium blue triangle and a small yellow circle. To the right of and above the small yellow circle there is the medium blue triangle. Block B has a big blue square, one medium black triangle and one medium yellow triangle. The medium yellow triangle is to the right of and above a small blue triangle. The big blue square is to the right of and above the medium yellow triangle. The small blue triangle is touching the bottom edge of this block.. The medium black triangle is near to and above the small blue triangle. And block C has one medium blue square.",
"question": "Is the medium blue square , below the small yellow object that is to the left of a black triangle?",
"choices": {
"Yes": "Yes",
"No": "No",
"DK": "DK"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | No | Incorrect |
| Opus 5.5 | DK | Correct |
| GLiDE v1.2 | DK | Correct |
| Decider v1.1 | DK | Correct |
| Jev | No | Incorrect |
| Microsoft Decision-1 | No | Incorrect |
| Clef | No | Incorrect |
| Luna Decisions | DK | Correct |
| Clef Flash | No | Incorrect |
| Strands Decider | No | Incorrect |
| GLiNER2.5 Decide | Yes | Incorrect |
| Laya | No | Incorrect |
| CalDec v1 | No | Incorrect |
Source and sharing
Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
"record": "6078/0",
"kind": "source_file_record"
},
"source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
"input_hash": "f94f686737e00b8cdef3ed3f62c8090fc152d356eaad2821c0944d4c3f7898f6"
}