RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 64

Back to Information seeking · All capabilities

Question ID: questbench/6e996ef768b20ef2c858
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Jen is planning to sell her root crops at the local farmer's market. She has 6 yams which can be sold at $1.5 each, 10 sweet potatoes that cost $2 each, and carrots which cost $1.25 each. She also has beets, but she decides to keep them for her own use, so they won't be sold. If she sells all the yams, sweet potatoes, and carrots, but none of the beets, how much will she earn?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • E: Number of carrots
  • G: Number of beets

Reference answer: Number of carrots (E)

Exact evaluated input as JSON
{
  "state": "Jen is planning to sell her root crops at the local farmer's market. She has 6 yams which can be sold at $1.5 each, 10 sweet potatoes that cost $2 each, and  carrots which cost $1.25 each. She also has  beets, but she decides to keep them for her own use, so they won't be sold. If she sells all the yams, sweet potatoes, and carrots, but none of the beets, how much will she earn?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "E": "Number of carrots",
    "G": "Number of beets"
  }
}

Recorded model answers

Model Answer Outcome
Astra Number of carrots Correct
Opus 5.5 Number of carrots Correct
GLiDE v1.2 Number of carrots Correct
Decider v1.1 Number of carrots Correct
Jev Number of carrots Correct
Microsoft Decision-1 Number of carrots Correct
Clef Number of carrots Correct
Luna Decisions Number of carrots Correct
Clef Flash Number of carrots Correct
Strands Decider Number of carrots Correct
GLiNER2.5 Decide Number of beets Incorrect
Laya Number of carrots Correct
CalDec v1 Number of beets Incorrect

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "14391",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "b5aebd1e4005d6fb584b78a0dc5aa475705ee52adafcd3dfdeb4a145b71e55ac"
}

Type to search questions, variants and documentation.