RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 67

Back to Information seeking · All capabilities

Question ID: questbench/7ee42f4b2058b3907c2f
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Alisa biked 12 miles per hour while carrying a backpack. Stanley biked at 10 miles per hour for 2.5 hours. How many miles did Alisa and Stanley bike in total, regardless of the weight of Alisa's backpack or the temperature during Stanley's ride?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • A2: Alisa's time (hours)
  • X1: Alisa's backpack weight (pounds)
  • X2: Temperature during Stanley's ride (degrees)

Reference answer: Alisa's time (hours) (A2)

Exact evaluated input as JSON
{
  "state": "Alisa biked 12 miles per hour while carrying a backpack. Stanley biked at 10 miles per hour for 2.5 hours. How many miles did Alisa and Stanley bike in total, regardless of the weight of Alisa's backpack or the temperature during Stanley's ride?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "A2": "Alisa's time (hours)",
    "X1": "Alisa's backpack weight (pounds)",
    "X2": "Temperature during Stanley's ride (degrees)"
  }
}

Recorded model answers

Model Answer Outcome
Astra Alisa's time (hours) Correct
Opus 5.5 Alisa's time (hours) Correct
GLiDE v1.2 Alisa's time (hours) Correct
Decider v1.1 Alisa's time (hours) Correct
Jev Alisa's time (hours) Correct
Microsoft Decision-1 Alisa's time (hours) Correct
Clef Alisa's time (hours) Correct
Luna Decisions Alisa's time (hours) Correct
Clef Flash Alisa's time (hours) Correct
Strands Decider Alisa's time (hours) Correct
GLiNER2.5 Decide Temperature during Stanley's ride (degrees) Incorrect
Laya Alisa's time (hours) Correct
CalDec v1 Alisa's time (hours) Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "8551",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "45a1b9d4ff610f781bafd9a9a0e6645327cf73e1c7358e8e8df738c2061740a4"
}

Type to search questions, variants and documentation.