RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 64

Back to Spatial cognition · All capabilities

Question ID: spartqa/fc7257c11e5c71fcb907
Source: SPARTQA
Input format: adapted choices

Explore this question’s altered versions

Context

We have three blocks, A, B and C. Block B is below blocks C and A. Block A contains one medium blue square. The medium blue square is below a medium yellow square. Block B has two medium black squares. Below medium black square number two there is medium black square number one. Block C contains a medium black square.

Question

Which blocks don't have any medium shape that is below a black square?

Answer choices

  • A: ["A"]
  • B: []
  • C: ["B"]
  • D: ["B", "C"]
  • E: ["A", "B", "C"]
  • F: ["A", "C"]
  • G: ["A", "B"]
  • H: ["C"]

Reference answer: ["A", "C"] (F)

Exact evaluated input as JSON
{
  "state": "We have three blocks, A, B and C. Block B is below blocks C and A. Block A contains one medium blue square. The medium blue square is below a medium yellow square. Block B has two medium black squares. Below medium black square number two there is medium black square number one. Block C contains a medium black square.",
  "question": "Which blocks don't have any medium shape that is below a black square?",
  "choices": {
    "A": "[\"A\"]",
    "B": "[]",
    "C": "[\"B\"]",
    "D": "[\"B\", \"C\"]",
    "E": "[\"A\", \"B\", \"C\"]",
    "F": "[\"A\", \"C\"]",
    "G": "[\"A\", \"B\"]",
    "H": "[\"C\"]"
  }
}

Recorded model answers

Model Answer Outcome
Astra ["A", "C"] Correct
Opus 5.5 ["A", "C"] Correct
GLiDE v1.2 ["A", "C"] Correct
Decider v1.1 ["A", "C"] Correct
Jev ["A", "C"] Correct
Microsoft Decision-1 ["A", "C"] Correct
Clef ["A", "C"] Correct
Luna Decisions ["A", "C"] Correct
Clef Flash ["A", "C"] Correct
Strands Decider ["A"] Incorrect
GLiNER2.5 Decide ["A"] Incorrect
Laya ["B", "C"] Incorrect
CalDec v1 ["B"] Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
    "record": "10509/6",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "a8b17309e0b3948b8d3d261961166140ab0c576218e4e346b3675c62f670924a"
}

Type to search questions, variants and documentation.