RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 62

Back to Spatial cognition · All capabilities

Question ID: spartqa/47abb58681d5f6f45324
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

There are three blocks, A, B and C. There are two medium blue squares in block A. Medium blue square number one is touching the bottom edge of this block. Below a medium yellow square is medium blue square number two. Medium blue square number one is below the medium yellow square and medium blue square number two. Then, there is block B which has one medium blue square. Block C is to the right of block B and below block A. It contains one medium blue square.

Question

Which object is above a medium blue square? medium blue square number two or the medium blue square that is in block B?

Answer choices

  • 0: medium blue square number two
  • 1: the medium blue square that is in block B
  • 2: both of them
  • 3: none of them

Reference answer: medium blue square number two (0)

Exact evaluated input as JSON
{
  "state": "There are three blocks, A, B and C. There are two medium blue squares in block A. Medium blue square number one is touching the bottom edge of this block. Below a medium yellow square is medium blue square number two. Medium blue square number one is below the medium yellow square and medium blue square number two. Then, there is block B which has one medium blue square. Block C is to the right of block B and below block A. It contains one medium blue square.",
  "question": "Which object is above a medium blue square? medium blue square number two or the medium blue square that is in block B?",
  "choices": {
    "0": "medium blue square  number two",
    "1": "the medium blue square  that is in block B",
    "2": "both of them",
    "3": "none of them"
  }
}

Recorded model answers

Model Answer Outcome
Astra medium blue square number two Correct
Opus 5.5 medium blue square number two Correct
GLiDE v1.2 medium blue square number two Correct
Decider v1.1 medium blue square number two Correct
Jev medium blue square number two Correct
Microsoft Decision-1 medium blue square number two Correct
Clef medium blue square number two Correct
Luna Decisions medium blue square number two Correct
Clef Flash medium blue square number two Correct
Strands Decider medium blue square number two Correct
GLiNER2.5 Decide the medium blue square that is in block B Incorrect
Laya both of them Incorrect
CalDec v1 the medium blue square that is in block B Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/test.json",
    "record": "637/3",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "d9a6cb5c40baab0f9a46a0293e63fa5bf9278e00619e7b46182ce6d3c0b7e5d1"
}

Type to search questions, variants and documentation.