RaindropA practical guide to choosing decision modelsGitHub ↗

Numerical cognition · question 26

Back to Numerical cognition · All capabilities

Question ID: bbh/7b7f2169639b9d218188
Source: BIG-Bench Hard selected tasks
Input format: adapted choices

Explore this question’s altered versions

Question

((-2 * -8 * 9 + -5) + (7 - -1 - -4 - 1)) =

Answer choices

  • A: 11
  • B: 16
  • C: 136
  • D: 150

Reference answer: 150 (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "((-2 * -8 * 9 + -5) + (7 - -1 - -4 - 1)) =",
  "choices": {
    "A": "11",
    "B": "16",
    "C": "136",
    "D": "150"
  }
}

Recorded model answers

Model Answer Outcome
Astra 150 Correct
Opus 5.5 150 Correct
GLiDE v1.2 136 Incorrect
Decider v1.1 136 Incorrect
Jev 136 Incorrect
Microsoft Decision-1 150 Correct
Clef 136 Incorrect
Luna Decisions 150 Correct
Clef Flash 16 Incorrect
Strands Decider 150 Correct
GLiNER2.5 Decide 11 Incorrect
Laya 11 Incorrect
CalDec v1 11 Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/multistep_arithmetic_two.json",
    "record": "43",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "11b91074b4a9b927a4e94328c5795c41a2fa5893cc29a4bcca8b71558a9779f7"
}

Type to search questions, variants and documentation.