RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 11

Back to Causal reasoning · All capabilities

Question ID: crass/c0dbf291e4d3da7afa58
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A woman dates a man. What would have happened if the woman had cheated on the man?

Answer choices

  • A: The man would be embarrassed to have been caught cheating.
  • B: That is not possible.
  • C: The man would have been very happy and would have married her.
  • D: Their relationship would have been put in jeopardy.

Reference answer: Their relationship would have been put in jeopardy. (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A woman dates a man. What would have happened if the woman had cheated on the man?",
  "choices": {
    "A": "The man would be embarrassed to have been caught cheating.",
    "B": "That is not possible.",
    "C": "The man would have been very happy and would have married her.",
    "D": "Their relationship would have been put in jeopardy."
  }
}

Recorded model answers

Model Answer Outcome
Astra Their relationship would have been put in jeopardy. Correct
Opus 5.5 Their relationship would have been put in jeopardy. Correct
GLiDE v1.2 Their relationship would have been put in jeopardy. Correct
Decider v1.1 Their relationship would have been put in jeopardy. Correct
Jev Their relationship would have been put in jeopardy. Correct
Microsoft Decision-1 Their relationship would have been put in jeopardy. Correct
Clef Their relationship would have been put in jeopardy. Correct
Luna Decisions Their relationship would have been put in jeopardy. Correct
Clef Flash Their relationship would have been put in jeopardy. Correct
Strands Decider Their relationship would have been put in jeopardy. Correct
GLiNER2.5 Decide Their relationship would have been put in jeopardy. Correct
Laya The man would be embarrassed to have been caught cheating. Incorrect
CalDec v1 The man would be embarrassed to have been caught cheating. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "32",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "df28a51900d8728631e4a761ef63182e91c55317f06668ddeb5dd6cce920023d"
}

Type to search questions, variants and documentation.