RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 58

Back to Spatial cognition · All capabilities

Question ID: spartqa/356448f6aa44b12229af
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

We have three blocks, A, B and C. Block A has one big black triangle. The big black triangle is touching the right edge of this block. This block also has a big yellow circle. Near to, to the right of, touching and above the big yellow circle there is the big black triangle. The big yellow circle is touching the bottom edge of this block. Then, we have block B. It has a medium black circle and a medium yellow square. The medium yellow square is to the right of and below a medium blue circle. The medium black circle is near to and above the medium blue circle. To the left of block A and above block B there is block C with a medium blue square and a big yellow square in it. Below the big yellow square there is the medium blue square.

Question

Is there a square that is below a blue square , to the right of a medium blue thing that is near to a black circle?

Answer choices

  • Yes: Yes
  • No: No
  • DK: DK

Reference answer: Yes (Yes)

Exact evaluated input as JSON
{
  "state": "We have three blocks, A, B and C. Block A has one big black triangle. The big black triangle is touching the right edge of this block. This block also has a big yellow circle. Near to, to the right of, touching and above the big yellow circle there is the big black triangle. The big yellow circle is touching the bottom edge of this block. Then, we have block B. It has a medium black circle and a medium yellow square. The medium yellow square is to the right of and below a medium blue circle. The medium black circle is near to and above the medium blue circle. To the left of block A and above block B there is block C with a medium blue square and a big yellow square in it. Below the big yellow square there is the medium blue square.",
  "question": "Is there a square that is below a blue square , to the right of a medium blue thing that is near to a black circle?",
  "choices": {
    "Yes": "Yes",
    "No": "No",
    "DK": "DK"
  }
}

Recorded model answers

Model Answer Outcome
Astra Yes Correct
Opus 5.5 Yes Correct
GLiDE v1.2 Yes Correct
Decider v1.1 Yes Correct
Jev No Incorrect
Microsoft Decision-1 No Incorrect
Clef No Incorrect
Luna Decisions No Incorrect
Clef Flash No Incorrect
Strands Decider Yes Correct
GLiNER2.5 Decide Yes Correct
Laya Yes Correct
CalDec v1 Yes Correct

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
    "record": "6316/1",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "6acfeeeaee39cfff87e65d9346bb566f657ccdf887e65bb566b2cf0e30e552eb"
}

Type to search questions, variants and documentation.