RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 52

Back to Spatial cognition · All capabilities

Question ID: spartqa/11c5e780667d92d6e901
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

There are two blocks. Lets call them A and B. Block A has three medium black squares. Below medium black square number three there is medium black square number two. Below a medium blue square there is medium black square number three. Medium black square number one is touching the bottom edge of this block. It is below the medium blue square , medium black square number three and medium black square number two. To the right of block A is block B. It contains one medium black square. Below a medium yellow square there is the medium black square.

Question

What is the relation between the square which is below a yellow thing and the medium black thing that is above medium square number two?

Answer choices

  • 0: left
  • 1: right
  • 2: above
  • 3: below
  • 4: near to
  • 5: far from
  • 6: touching
  • 7: DK

Reference answer: right (1)

Exact evaluated input as JSON
{
  "state": "There are two blocks. Lets call them A and B. Block A has three medium black squares. Below medium black square number three there is medium black square number two. Below a medium blue square there is medium black square number three. Medium black square number one is touching the bottom edge of this block. It is below the medium blue square , medium black square number three and medium black square number two. To the right of block A is block B. It contains one medium black square. Below a medium yellow square there is the medium black square.",
  "question": "What is the relation between the square which is below a yellow thing and the medium black thing that is above medium square number two?",
  "choices": {
    "0": "left",
    "1": "right",
    "2": "above",
    "3": "below",
    "4": "near to",
    "5": "far from",
    "6": "touching",
    "7": "DK"
  }
}

Recorded model answers

Model Answer Outcome
Astra right Correct
Opus 5.5 right Correct
GLiDE v1.2 right Correct
Decider v1.1 right Correct
Jev right Correct
Microsoft Decision-1 right Correct
Clef far from Incorrect
Luna Decisions DK Incorrect
Clef Flash right Correct
Strands Decider below Incorrect
GLiNER2.5 Decide touching Incorrect
Laya above Incorrect
CalDec v1 left Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/test.json",
    "record": "321/5",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "5bba415257da36e7fd4a8332fed1c1502dc184a31efe4fabcf0b4e4ad2537558"
}

Type to search questions, variants and documentation.