RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 9

Back to Causal reasoning · All capabilities

Question ID: crass/bb5de0e9c0915ff53920
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A boy strikes a match. What would have happened if the boy had ditched the match?

Answer choices

  • A: The lighting of the match would not have occurred.
  • B: He would have been disqualified.
  • C: I don't know.
  • D: The game would have ended tied.

Reference answer: The lighting of the match would not have occurred. (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A boy strikes a match. What would have happened if the boy had ditched the match?",
  "choices": {
    "A": "The lighting of the match would not have occurred.",
    "B": "He would have been disqualified.",
    "C": "I don't know.",
    "D": "The game would have ended tied."
  }
}

Recorded model answers

Model Answer Outcome
Astra The lighting of the match would not have occurred. Correct
Opus 5.5 The lighting of the match would not have occurred. Correct
GLiDE v1.2 The lighting of the match would not have occurred. Correct
Decider v1.1 The lighting of the match would not have occurred. Correct
Jev The lighting of the match would not have occurred. Correct
Microsoft Decision-1 The lighting of the match would not have occurred. Correct
Clef The lighting of the match would not have occurred. Correct
Luna Decisions The lighting of the match would not have occurred. Correct
Clef Flash The lighting of the match would not have occurred. Correct
Strands Decider The lighting of the match would not have occurred. Correct
GLiNER2.5 Decide He would have been disqualified. Incorrect
Laya The lighting of the match would not have occurred. Correct
CalDec v1 The lighting of the match would not have occurred. Correct

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "38",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "affd32103eaf2a11fa47e8ee8aa9f1e091f3b56e0f3b9eeefcd3e9e5f04c7918"
}

Type to search questions, variants and documentation.