RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 28

Back to Spatial cognition · All capabilities

Question ID: spartqa/d4e57fb981afcb09c105
Source: SPARTQA
Input format: adapted choices

Explore this question’s altered versions

Context

There are two blocks. We call them A and B. Block A is to the left of B. Block A contains two medium blue squares. Medium blue square number one is below medium blue square number two. Block B has a medium yellow square and a medium black square. The medium yellow square is below the medium black square. There is also a medium blue square in this block. Below the medium black square and the medium yellow square is the medium blue square.

Question

Which blocks have a square that is above a medium blue shape?

Answer choices

  • A: ["A", "B"]
  • B: ["B"]
  • C: ["C"]
  • D: ["A", "B", "C"]
  • E: []
  • F: ["A"]
  • G: ["A", "C"]
  • H: ["B", "C"]

Reference answer: ["A", "B"] (A)

Exact evaluated input as JSON
{
  "state": "There are two blocks. We call them A and B. Block A is to the left of B. Block A contains two medium blue squares. Medium blue square number one is below medium blue square number two. Block B has a medium yellow square and a medium black square. The medium yellow square is below the medium black square. There is also a medium blue square in this block. Below the medium black square and the medium yellow square is the medium blue square.",
  "question": "Which blocks have a square that is above a medium blue shape?",
  "choices": {
    "A": "[\"A\", \"B\"]",
    "B": "[\"B\"]",
    "C": "[\"C\"]",
    "D": "[\"A\", \"B\", \"C\"]",
    "E": "[]",
    "F": "[\"A\"]",
    "G": "[\"A\", \"C\"]",
    "H": "[\"B\", \"C\"]"
  }
}

Recorded model answers

Model Answer Outcome
Astra ["A", "B"] Correct
Opus 5.5 ["A", "B"] Correct
GLiDE v1.2 ["A", "B"] Correct
Decider v1.1 ["A", "B"] Correct
Jev ["A", "B"] Correct
Microsoft Decision-1 ["A", "B"] Correct
Clef ["A", "B"] Correct
Luna Decisions ["A", "B"] Correct
Clef Flash ["A", "B"] Correct
Strands Decider ["A", "B"] Correct
GLiNER2.5 Decide ["A", "B"] Correct
Laya ["B"] Incorrect
CalDec v1 ["B"] Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
    "record": "7599/7",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "421b3c89a067efdbffa39598c4f44223edbac76ea4775490fbb560e5b0c7fe6a"
}

Type to search questions, variants and documentation.