RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 17

Back to Causal reasoning · All capabilities

Question ID: crass/d207c42752bd0975a8b8
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A woman opens a treasure chest. What would have happened if the woman had not opened the treasure chest?

Answer choices

  • A: The treasure chest would have been open.
  • B: That is not possible.
  • C: The treasure chest would have remained closed.
  • D: I don't know.

Reference answer: The treasure chest would have remained closed. (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A woman opens a treasure chest. What would have happened if the woman had not opened the treasure chest?",
  "choices": {
    "A": "The treasure chest would have been open.",
    "B": "That is not possible.",
    "C": "The treasure chest would have remained closed.",
    "D": "I don't know."
  }
}

Recorded model answers

Model Answer Outcome
Astra The treasure chest would have remained closed. Correct
Opus 5.5 The treasure chest would have remained closed. Correct
GLiDE v1.2 The treasure chest would have remained closed. Correct
Decider v1.1 The treasure chest would have remained closed. Correct
Jev The treasure chest would have remained closed. Correct
Microsoft Decision-1 The treasure chest would have remained closed. Correct
Clef The treasure chest would have remained closed. Correct
Luna Decisions The treasure chest would have remained closed. Correct
Clef Flash The treasure chest would have remained closed. Correct
Strands Decider The treasure chest would have remained closed. Correct
GLiNER2.5 Decide The treasure chest would have been open. Incorrect
Laya The treasure chest would have been open. Incorrect
CalDec v1 The treasure chest would have been open. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "0",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "1f2da541cb25404ef20fb9988d466c2194e1d09ddd0eba3e0a23c35458d47165"
}

Type to search questions, variants and documentation.