RaindropA practical guide to choosing decision modelsGitHub ↗

Numerical cognition · question 55

Back to Numerical cognition · All capabilities

Question ID: bbh/a6e2d5a7063d698564f3
Source: BIG-Bench Hard selected tasks
Input format: adapted choices

Explore this question’s altered versions

Question

((-7 + 9 + -9 * -4) - (-7 * 6 - 7 * 3)) =

Answer choices

  • A: 17
  • B: 101
  • C: 1
  • D: 31

Reference answer: 101 (B)

Exact evaluated input as JSON
{
  "state": "",
  "question": "((-7 + 9 + -9 * -4) - (-7 * 6 - 7 * 3)) =",
  "choices": {
    "A": "17",
    "B": "101",
    "C": "1",
    "D": "31"
  }
}

Recorded model answers

Model Answer Outcome
Astra 101 Correct
Opus 5.5 101 Correct
GLiDE v1.2 101 Correct
Decider v1.1 101 Correct
Jev 101 Correct
Microsoft Decision-1 31 Incorrect
Clef 31 Incorrect
Luna Decisions 101 Correct
Clef Flash 31 Incorrect
Strands Decider 31 Incorrect
GLiNER2.5 Decide 17 Incorrect
Laya 17 Incorrect
CalDec v1 17 Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/multistep_arithmetic_two.json",
    "record": "157",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "3b1989c302be06737738c7b34fe3fbd27297f9ab6816d614abf583bdb0acef62"
}

Type to search questions, variants and documentation.