RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 7

Back to Information seeking · All capabilities

Question ID: questbench/cf62fc49628fdf24b038
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Heather bought a jumbo bag of raisins. She wants to make oatmeal cookies, granola, and a snack mix with the raisins. A batch of oatmeal cookies takes 3/4 of a cup of raisins. She also bought a jumbo bag of raisins to make small cakes, each of which required few cups of raisins. How many batches of oatmeal cookies can Heather make if she divides the bag of raisins equally among the cookies, granola, and snack mix?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • T: Total cups of raisins in the jumbo bag
  • F2: Cups of raisins required for one small cake

Reference answer: Total cups of raisins in the jumbo bag (T)

Exact evaluated input as JSON
{
  "state": "Heather bought a jumbo bag of raisins. She wants to make oatmeal cookies, granola, and a snack mix with the raisins. A batch of oatmeal cookies takes 3/4 of a cup of raisins. She also bought a jumbo bag of raisins to make small cakes, each of which required few cups of raisins. How many batches of oatmeal cookies can Heather make if she divides the bag of raisins equally among the cookies, granola, and snack mix?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "T": "Total cups of raisins in the jumbo bag",
    "F2": "Cups of raisins required for one small cake"
  }
}

Recorded model answers

Model Answer Outcome
Astra Total cups of raisins in the jumbo bag Correct
Opus 5.5 Total cups of raisins in the jumbo bag Correct
GLiDE v1.2 Total cups of raisins in the jumbo bag Correct
Decider v1.1 Total cups of raisins in the jumbo bag Correct
Jev Total cups of raisins in the jumbo bag Correct
Microsoft Decision-1 Total cups of raisins in the jumbo bag Correct
Clef Total cups of raisins in the jumbo bag Correct
Luna Decisions Total cups of raisins in the jumbo bag Correct
Clef Flash Total cups of raisins in the jumbo bag Correct
Strands Decider Total cups of raisins in the jumbo bag Correct
GLiNER2.5 Decide Cups of raisins required for one small cake Incorrect
Laya Total cups of raisins in the jumbo bag Correct
CalDec v1 Total cups of raisins in the jumbo bag Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "14820",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "1cfad2e6f6eef71617ae9a4ff9984dcf2ab9c018b17bbbbf74c293c669f5f2ce"
}

Type to search questions, variants and documentation.