RaindropA practical guide to choosing decision modelsGitHub ↗

Causal reasoning · question 5

Back to Causal reasoning · All capabilities

Question ID: crass/11458eb4c332c5c3323c
Source: CRASS
Input format: native choices

Explore this question’s altered versions

Question

A politician accepts an invitation. What would have happened if the politician had accepted a bribe?

Answer choices

  • A: Politicians accept invitations all the time.
  • B: Politicians should accept invitations and bribes.
  • C: They would have been congratulated.
  • D: If anyone found out, the politician would have gotten in trouble and lost their office.

Reference answer: If anyone found out, the politician would have gotten in trouble and lost their office. (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A politician accepts an invitation. What would have happened if the politician had accepted a bribe?",
  "choices": {
    "A": "Politicians accept invitations all the time.",
    "B": "Politicians should accept invitations and bribes.",
    "C": "They would have been congratulated.",
    "D": "If anyone found out, the politician would have gotten in trouble and lost their office."
  }
}

Recorded model answers

Model Answer Outcome
Astra If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Opus 5.5 If anyone found out, the politician would have gotten in trouble and lost their office. Correct
GLiDE v1.2 If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Decider v1.1 If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Jev If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Microsoft Decision-1 If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Clef If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Luna Decisions If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Clef Flash If anyone found out, the politician would have gotten in trouble and lost their office. Correct
Strands Decider If anyone found out, the politician would have gotten in trouble and lost their office. Correct
GLiNER2.5 Decide Politicians should accept invitations and bribes. Incorrect
Laya Politicians accept invitations all the time. Incorrect
CalDec v1 Politicians accept invitations all the time. Incorrect

Source and sharing

Attribution: BIG-bench crass_ai contributors.
Terms: Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bigbench/benchmark_tasks/crass_ai/task.json",
    "record": "34",
    "kind": "source_file_record"
  },
  "source_revision": "092b196c1f8f14a54bbc62f24759d43bde46dd3b",
  "input_hash": "33aa7766b10a561457a450731c4b74a695743bfd3f627c6cf79225422ecc4b8e"
}

Type to search questions, variants and documentation.