RaindropA practical guide to choosing decision modelsGitHub ↗

Numerical cognition · question 66

Back to Numerical cognition · All capabilities

Question ID: bbh/e458b6c04bf1aa068f09
Source: BIG-Bench Hard selected tasks
Input format: adapted choices

Explore this question’s altered versions

Question

((-4 + -6 + -6 - -9) + (-2 * -5 * 5 * 7)) =

Answer choices

  • A: 343
  • B: -16
  • C: 325
  • D: 50

Reference answer: 343 (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "((-4 + -6 + -6 - -9) + (-2 * -5 * 5 * 7)) =",
  "choices": {
    "A": "343",
    "B": "-16",
    "C": "325",
    "D": "50"
  }
}

Recorded model answers

Model Answer Outcome
Astra 343 Correct
Opus 5.5 343 Correct
GLiDE v1.2 325 Incorrect
Decider v1.1 325 Incorrect
Jev 343 Correct
Microsoft Decision-1 325 Incorrect
Clef 325 Incorrect
Luna Decisions 325 Incorrect
Clef Flash 325 Incorrect
Strands Decider 325 Incorrect
GLiNER2.5 Decide 343 Correct
Laya 343 Correct
CalDec v1 343 Correct

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/multistep_arithmetic_two.json",
    "record": "136",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "9bb4c0eda60598f8e228043a294de7443499bb623b4b3fa16359110cc56fa9f0"
}

Type to search questions, variants and documentation.