Information seeking · question 7
Back to Information seeking · All capabilities
Question ID: questbench/cf62fc49628fdf24b038
Source: QuestBench (Google DeepMind)
Input format: native choices
Explore this question’s altered versions
Context
Heather bought a jumbo bag of raisins. She wants to make oatmeal cookies, granola, and a snack mix with the raisins. A batch of oatmeal cookies takes 3/4 of a cup of raisins. She also bought a jumbo bag of raisins to make small cakes, each of which required few cups of raisins. How many batches of oatmeal cookies can Heather make if she divides the bag of raisins equally among the cookies, granola, and snack mix?
Question
Which missing variable should be asked about to solve this problem?
Answer choices
- T: Total cups of raisins in the jumbo bag
- F2: Cups of raisins required for one small cake
Reference answer: Total cups of raisins in the jumbo bag (T)
Exact evaluated input as JSON
{
"state": "Heather bought a jumbo bag of raisins. She wants to make oatmeal cookies, granola, and a snack mix with the raisins. A batch of oatmeal cookies takes 3/4 of a cup of raisins. She also bought a jumbo bag of raisins to make small cakes, each of which required few cups of raisins. How many batches of oatmeal cookies can Heather make if she divides the bag of raisins equally among the cookies, granola, and snack mix?",
"question": "Which missing variable should be asked about to solve this problem?",
"choices": {
"T": "Total cups of raisins in the jumbo bag",
"F2": "Cups of raisins required for one small cake"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Total cups of raisins in the jumbo bag | Correct |
| Opus 5.5 | Total cups of raisins in the jumbo bag | Correct |
| GLiDE v1.2 | Total cups of raisins in the jumbo bag | Correct |
| Decider v1.1 | Total cups of raisins in the jumbo bag | Correct |
| Jev | Total cups of raisins in the jumbo bag | Correct |
| Microsoft Decision-1 | Total cups of raisins in the jumbo bag | Correct |
| Clef | Total cups of raisins in the jumbo bag | Correct |
| Luna Decisions | Total cups of raisins in the jumbo bag | Correct |
| Clef Flash | Total cups of raisins in the jumbo bag | Correct |
| Strands Decider | Total cups of raisins in the jumbo bag | Correct |
| GLiNER2.5 Decide | Cups of raisins required for one small cake | Incorrect |
| Laya | Total cups of raisins in the jumbo bag | Correct |
| CalDec v1 | Total cups of raisins in the jumbo bag | Correct |
Source and sharing
Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
"record": "14820",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "1cfad2e6f6eef71617ae9a4ff9984dcf2ab9c018b17bbbbf74c293c669f5f2ce"
}