RaindropA practical guide to choosing decision modelsGitHub ↗

Spatial cognition · question 8

Back to Spatial cognition · All capabilities

Question ID: spartqa/707747556e215d5f2917
Source: SPARTQA
Input format: native choices

Explore this question’s altered versions

Context

There are three blocks, A, B and C. Block C is to the right of and B is to the left of A. Block A has a medium black circle. Block B has a medium blue square. Block C has one medium blue circle, one small black triangle and a small yellow triangle. The medium blue circle is touching the right edge of this block. To the left of and below the small yellow triangle is the small black triangle. The medium blue circle is to the right of the small yellow triangle. It is above the small black triangle.

Question

Which object is to the right of a square? the small yellow triangle or the small black triangle?

Answer choices

  • 0: the small yellow triangle
  • 1: the small black triangle
  • 2: both of them
  • 3: none of them

Reference answer: both of them (2)

Exact evaluated input as JSON
{
  "state": "There are three blocks, A, B and C. Block C is to the right of and B is to the left of A. Block A has a medium black circle. Block B has a medium blue square. Block C has one medium blue circle, one small black triangle and a small yellow triangle. The medium blue circle is touching the right edge of this block. To the left of and below the small yellow triangle is the small black triangle. The medium blue circle is to the right of the small yellow triangle. It is above the small black triangle.",
  "question": "Which object is to the right of a square? the small yellow triangle or the small black triangle?",
  "choices": {
    "0": "the small yellow triangle ",
    "1": "the small black triangle ",
    "2": "both of them",
    "3": "none of them"
  }
}

Recorded model answers

Model Answer Outcome
Astra both of them Correct
Opus 5.5 both of them Correct
GLiDE v1.2 both of them Correct
Decider v1.1 both of them Correct
Jev both of them Correct
Microsoft Decision-1 both of them Correct
Clef both of them Correct
Luna Decisions the small yellow triangle Incorrect
Clef Flash both of them Correct
Strands Decider both of them Correct
GLiNER2.5 Decide the small black triangle Incorrect
Laya both of them Correct
CalDec v1 the small black triangle Incorrect

Source and sharing

Attribution: HLR / Roshanak Mirzaee et al..
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/SpartQA_Auto.zip/SpartQA_Auto/dev.json",
    "record": "1069/3",
    "kind": "source_file_record"
  },
  "source_revision": "9892ed1bff9103f5a9103c68bd271b7630ab661e",
  "input_hash": "83e68120b23985a0b47683bc71a7fe67ff263bf9128a57aabb6bd2638473c117"
}

Type to search questions, variants and documentation.