RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 66

Back to Spatial cognition · All capabilities

Question ID: spartqa/6d1fbdd11b3d5316f8de
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

We have three blocks. We call them A, B and C. Block B is to the left of C and block C is above A. Block A contains one medium yellow circle, one big yellow triangle and one small blue circle. Above a big blue square there is the medium yellow circle. The big yellow triangle is above the small blue circle and the big blue square. Near to the big blue square there is the small blue circle. The medium yellow circle is to the left of the yellow object which is above the small blue circle. The small blue circle is touching the bottom edge of this block. Block B contains one small blue circle. Block C contains two big yellow triangles. Big yellow triangle number one is touching the right edge of this block. Big yellow triangle number two is touching the right edge of this block. There is also one big blue square in this block. It is near to big yellow triangle number two. Below big yellow triangle number two and the big blue square there is big yellow triangle number one.

Question

What is the relation between the square which is above big yellow triangle number one and the yellow triangle that is above a big yellow triangle that is touching the right edge of a block?

Answer choices

  • 0: left
  • 1: right
  • 2: above
  • 3: below
  • 4: near to
  • 5: far from
  • 6: touching
  • 7: DK

Reference answer: near to (4)

Exact evaluated input as JSON
{
  "state": "We have three blocks. We call them A, B and C. Block B is to the left of C and block C is above A. Block A contains one medium yellow circle, one big yellow triangle and one small blue circle. Above a big blue square there is the medium yellow circle. The big yellow triangle is above the small blue circle and the big blue square. Near to the big blue square there is the small blue circle. The medium yellow circle is to the left of the yellow object which is above the small blue circle. The small blue circle is touching the bottom edge of this block. Block B contains one small blue circle. Block C contains two big yellow triangles. Big yellow triangle number one is touching the right edge of this block. Big yellow triangle number two is touching the right edge of this block. There is also one big blue square in this block. It is near to big yellow triangle number two. Below big yellow triangle number two and the big blue square there is big yellow triangle number one.",
  "question": "What is the relation between the square which is above big yellow triangle number one and the yellow triangle that is above a big yellow triangle that is touching the right edge of a block?",
  "choices": {
    "0": "left",
    "1": "right",
    "2": "above",
    "3": "below",
    "4": "near to",
    "5": "far from",
    "6": "touching",
    "7": "DK"
  }
}

Recorded model answers

Model Answer Outcome
Astra near to Correct
Opus 5.5 near to Correct
GLiDE v1.2 near to Correct
Decider v1.1 near to Correct
Jev near to Correct
Microsoft Decision-1 near to Correct
Clef near to Correct
Luna Decisions DK Incorrect
Clef Flash right Incorrect
Strands Decider near to Correct
GLiNER2.5 Decide touching Incorrect
Laya below Incorrect
CalDec v1 below Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
    "record": "4467/5",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "0b5c8ee97b14771058f96b3187c42b475cfd5442b3e4d27c21f9141fb21d2b6d"
}

Type to search questions, variants and documentation.