RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 2

Back to Causal reasoning · All capabilities

Question ID: crass/24894f4e703cea94f7f8
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A stone hits a window. What would have happened if the stone had flown by the window?

Answer choices

  • A: The window would have stayed intact.
  • B: I would have been hit by a stone.
  • C: That is not possible.
  • D: The window would have been shattered.

Reference answer: The window would have stayed intact. (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A stone hits a window. What would have happened if the stone had flown by the window?",
  "choices": {
    "A": "The window would have stayed intact.",
    "B": "I would have been hit by a stone.",
    "C": "That is not possible.",
    "D": "The window would have been shattered."
  }
}

Recorded model answers

Model Answer Outcome
Astra The window would have stayed intact. Correct
Opus 5.5 The window would have stayed intact. Correct
GLiDE v1.2 The window would have stayed intact. Correct
Decider v1.1 The window would have stayed intact. Correct
Jev The window would have stayed intact. Correct
Microsoft Decision-1 The window would have stayed intact. Correct
Clef The window would have stayed intact. Correct
Luna Decisions The window would have stayed intact. Correct
Clef Flash The window would have stayed intact. Correct
Strands Decider The window would have been shattered. Incorrect
GLiNER2.5 Decide The window would have been shattered. Incorrect
Laya The window would have been shattered. Incorrect
CalDec v1 I would have been hit by a stone. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "40",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "a63858a7d12a1d1869f294769bbecd2bae9666885d9d7777c60cb182ecead49f"
}

Type to search questions, variants and documentation.