RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 54

Back to Spatial cognition · All capabilities

Question ID: spartqa/12df91f6e1630eeec74a
Source: SPARTQA
Input format: adapted choices

Explore this question’s altered versions

Context

We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.

Question

Which blocks don't have any medium square that is below a medium square?

Answer choices

  • A: ["A", "C"]
  • B: ["A", "B", "C"]
  • C: ["B", "C"]
  • D: ["A"]
  • E: ["C"]
  • F: ["B"]
  • G: ["A", "B"]
  • H: []

Reference answer: ["C"] (E)

Exact evaluated input as JSON
{
  "state": "We have three blocks, A, B and C. Block A is below block C and above block B. Block A has two medium yellow squares. Medium yellow square number one is touching the bottom edge of this block. This shape is below medium yellow square number two. Block B contains one medium blue square. One medium yellow square is also in this block. The medium blue square is below the medium yellow square. And block C contains one medium blue square.",
  "question": "Which blocks don't have any medium square that is below a medium square?",
  "choices": {
    "A": "[\"A\", \"C\"]",
    "B": "[\"A\", \"B\", \"C\"]",
    "C": "[\"B\", \"C\"]",
    "D": "[\"A\"]",
    "E": "[\"C\"]",
    "F": "[\"B\"]",
    "G": "[\"A\", \"B\"]",
    "H": "[]"
  }
}

Recorded model answers

Model Answer Outcome
Astra ["C"] Correct
Opus 5.5 ["C"] Correct
GLiDE v1.2 ["C"] Correct
Decider v1.1 ["C"] Correct
Jev ["C"] Correct
Microsoft Decision-1 ["C"] Correct
Clef ["C"] Correct
Luna Decisions ["A", "C"] Incorrect
Clef Flash ["C"] Correct
Strands Decider ["A", "C"] Incorrect
GLiNER2.5 Decide ["A", "C"] Incorrect
Laya ["A", "B", "C"] Incorrect
CalDec v1 ["A", "C"] Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/dev.json",
    "record": "29/6",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "f95adad07f626deff421237e34a1bcaa330d12d84f9eef517aff1da3e8ccdca3"
}

Type to search questions, variants and documentation.