RaindropA practical guide to choosing decision modelsGitHub ↗

Physical cognition · question 4

Back to Physical cognition · All capabilities

Question ID: crass/0eb5b74a6acc583ad59a
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A man catches a ball. What would have happened if the man had caught a refrigerator?

Answer choices

  • A: He would have got hurt.
  • B: That man would have been a refrigerator.
  • C: He would have thrown it back.
  • D: Everything would have been fine.

Reference answer: He would have got hurt. (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A man catches a ball. What would have happened if the man had caught a refrigerator?",
  "choices": {
    "A": "He would have got hurt.",
    "B": "That man would have been a refrigerator.",
    "C": "He would have thrown it back.",
    "D": "Everything would have been fine."
  }
}

Recorded model answers

Model Answer Outcome
Astra He would have got hurt. Correct
Opus 5.5 He would have got hurt. Correct
GLiDE v1.2 He would have got hurt. Correct
Decider v1.1 He would have got hurt. Correct
Jev He would have got hurt. Correct
Microsoft Decision-1 He would have got hurt. Correct
Clef He would have got hurt. Correct
Luna Decisions He would have got hurt. Correct
Clef Flash He would have got hurt. Correct
Strands Decider He would have got hurt. Correct
GLiNER2.5 Decide That man would have been a refrigerator. Incorrect
Laya That man would have been a refrigerator. Incorrect
CalDec v1 He would have thrown it back. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "17",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "f7de15f95722e0deefe3a82386c77a578b5f9625334b162bdb9d072cd710c80c"
}

Type to search questions, variants and documentation.