RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 50

Back to Spatial cognition · All capabilities

Question ID: spartqa/5c9863f43a411486d1b2
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

There are three blocks. Lets call them A, B and C. Block B is to the right of A and block C is above B. Block A has one medium blue triangle and a small yellow circle. To the right of and above the small yellow circle there is the medium blue triangle. Block B has a big blue square, one medium black triangle and one medium yellow triangle. The medium yellow triangle is to the right of and above a small blue triangle. The big blue square is to the right of and above the medium yellow triangle. The small blue triangle is touching the bottom edge of this block.. The medium black triangle is near to and above the small blue triangle. And block C has one medium blue square.

Question

Is the medium blue square , below the small yellow object that is to the left of a black triangle?

Answer choices

  • Yes: Yes
  • No: No
  • DK: DK

Reference answer: DK (DK)

Exact evaluated input as JSON
{
  "state": "There are three blocks. Lets call them A, B and C. Block B is to the right of A and block C is above B. Block A has one medium blue triangle and a small yellow circle. To the right of and above the small yellow circle there is the medium blue triangle. Block B has a big blue square, one medium black triangle and one medium yellow triangle. The medium yellow triangle is to the right of and above a small blue triangle. The big blue square is to the right of and above the medium yellow triangle. The small blue triangle is touching the bottom edge of this block.. The medium black triangle is near to and above the small blue triangle. And block C has one medium blue square.",
  "question": "Is the medium blue square , below the small yellow object that is to the left of a black triangle?",
  "choices": {
    "Yes": "Yes",
    "No": "No",
    "DK": "DK"
  }
}

Recorded model answers

Model Answer Outcome
Astra No Incorrect
Opus 5.5 DK Correct
GLiDE v1.2 DK Correct
Decider v1.1 DK Correct
Jev No Incorrect
Microsoft Decision-1 No Incorrect
Clef No Incorrect
Luna Decisions DK Correct
Clef Flash No Incorrect
Strands Decider No Incorrect
GLiNER2.5 Decide Yes Incorrect
Laya No Incorrect
CalDec v1 No Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/train.json",
    "record": "6078/0",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "f94f686737e00b8cdef3ed3f62c8090fc152d356eaad2821c0944d4c3f7898f6"
}

Type to search questions, variants and documentation.