RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 11

Back to Information seeking · All capabilities

Question ID: questbench/3486876017d4e56ed279
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Leah just got a new kitten. Spaying her cost $200, her 3 vaccines costs $20 each, and the kitten broke vases that cost $12 each. Leah also bought a cat bed and a scratching post, but she used a gift card to pay for these items. How much has the kitten cost so far, excluding the items paid for with the gift card?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • D: number of vase
  • F: cat bed
  • G: scratching post

Reference answer: number of vase (D)

Exact evaluated input as JSON
{
  "state": "Leah just got a new kitten. Spaying her cost $200, her 3 vaccines costs $20 each, and the kitten broke vases that cost $12 each. Leah also bought a cat bed and a scratching post, but she used a gift card to pay for these items. How much has the kitten cost so far, excluding the items paid for with the gift card?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "D": "number of vase",
    "F": "cat bed",
    "G": "scratching post"
  }
}

Recorded model answers

Model Answer Outcome
Astra number of vase Correct
Opus 5.5 number of vase Correct
GLiDE v1.2 number of vase Correct
Decider v1.1 number of vase Correct
Jev number of vase Correct
Microsoft Decision-1 number of vase Correct
Clef number of vase Correct
Luna Decisions number of vase Correct
Clef Flash number of vase Correct
Strands Decider number of vase Correct
GLiNER2.5 Decide scratching post Incorrect
Laya number of vase Correct
CalDec v1 scratching post Incorrect

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "19791",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "a216780b60c02c99f1dcd71c30fdef8da0a19a2391fa96b53ed346fd4d78d296"
}

Type to search questions, variants and documentation.