Information seeking · question 11
Back to Information seeking · All capabilities
Question ID: questbench/3486876017d4e56ed279
Source: QuestBench (Google DeepMind)
Input format: native choices
Explore this question’s altered versions
Context
Leah just got a new kitten. Spaying her cost $200, her 3 vaccines costs $20 each, and the kitten broke vases that cost $12 each. Leah also bought a cat bed and a scratching post, but she used a gift card to pay for these items. How much has the kitten cost so far, excluding the items paid for with the gift card?
Question
Which missing variable should be asked about to solve this problem?
Answer choices
- D: number of vase
- F: cat bed
- G: scratching post
Reference answer: number of vase (D)
Exact evaluated input as JSON
{
"state": "Leah just got a new kitten. Spaying her cost $200, her 3 vaccines costs $20 each, and the kitten broke vases that cost $12 each. Leah also bought a cat bed and a scratching post, but she used a gift card to pay for these items. How much has the kitten cost so far, excluding the items paid for with the gift card?",
"question": "Which missing variable should be asked about to solve this problem?",
"choices": {
"D": "number of vase",
"F": "cat bed",
"G": "scratching post"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | number of vase | Correct |
| Opus 5.5 | number of vase | Correct |
| GLiDE v1.2 | number of vase | Correct |
| Decider v1.1 | number of vase | Correct |
| Jev | number of vase | Correct |
| Microsoft Decision-1 | number of vase | Correct |
| Clef | number of vase | Correct |
| Luna Decisions | number of vase | Correct |
| Clef Flash | number of vase | Correct |
| Strands Decider | number of vase | Correct |
| GLiNER2.5 Decide | scratching post | Incorrect |
| Laya | number of vase | Correct |
| CalDec v1 | scratching post | Incorrect |
Source and sharing
Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
"record": "19791",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "a216780b60c02c99f1dcd71c30fdef8da0a19a2391fa96b53ed346fd4d78d296"
}