RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 7

Back to Causal reasoning · All capabilities

Question ID: crass/a2b38c39eb86e3fdb6f2
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A car crashes into a tree. What would have happened if the car had parked beneath the tree?

Answer choices

  • A: The car would have been hit by the tree.
  • B: That is not possible.
  • C: I think it would have crashed into the tree.
  • D: Nothing special would have happened.

Reference answer: Nothing special would have happened. (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A car crashes into a tree. What would have happened if the car had parked beneath the tree?",
  "choices": {
    "A": "The car would have been hit by the tree.",
    "B": "That is not possible.",
    "C": "I think it would have crashed into the tree.",
    "D": "Nothing special would have happened."
  }
}

Recorded model answers

Model Answer Outcome
Astra Nothing special would have happened. Correct
Opus 5.5 Nothing special would have happened. Correct
GLiDE v1.2 Nothing special would have happened. Correct
Decider v1.1 Nothing special would have happened. Correct
Jev Nothing special would have happened. Correct
Microsoft Decision-1 Nothing special would have happened. Correct
Clef Nothing special would have happened. Correct
Luna Decisions Nothing special would have happened. Correct
Clef Flash Nothing special would have happened. Correct
Strands Decider That is not possible. Incorrect
GLiNER2.5 Decide The car would have been hit by the tree. Incorrect
Laya The car would have been hit by the tree. Incorrect
CalDec v1 The car would have been hit by the tree. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "39",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "f6230929104132c9c6c25b92699c2f5d65019a2b2ecc994ad46e113d85bd5dcc"
}

Type to search questions, variants and documentation.