RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 60

Back to Information seeking · All capabilities

Question ID: questbench/70f3e0ce25a345f3b019
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

In a family, there are 2 brothers and 3 sisters. All sisters are the same age. One of the brothers is 12 years old, which is half the age of the older brother. The family also has a pet dog that is years old and a pet parrot that is years old. What is the total age of all these siblings, not including the ages of the pets?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • B: SECOND SISTER'S AGE
  • F: PET DOG'S AGE
  • G: PET PARROT'S AGE

Reference answer: SECOND SISTER'S AGE (B)

Exact evaluated input as JSON
{
  "state": "In a family, there are 2 brothers and 3 sisters. All sisters are the same age. One of the brothers is 12 years old, which is half the age of the older brother. The family also has a pet dog that is years old and a pet parrot that is years old. What is the total age of all these siblings, not including the ages of the pets?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "B": "SECOND SISTER'S AGE",
    "F": "PET DOG'S AGE",
    "G": "PET PARROT'S AGE"
  }
}

Recorded model answers

Model Answer Outcome
Astra SECOND SISTER'S AGE Correct
Opus 5.5 SECOND SISTER'S AGE Correct
GLiDE v1.2 SECOND SISTER'S AGE Correct
Decider v1.1 SECOND SISTER'S AGE Correct
Jev SECOND SISTER'S AGE Correct
Microsoft Decision-1 SECOND SISTER'S AGE Correct
Clef SECOND SISTER'S AGE Correct
Luna Decisions SECOND SISTER'S AGE Correct
Clef Flash SECOND SISTER'S AGE Correct
Strands Decider SECOND SISTER'S AGE Correct
GLiNER2.5 Decide PET DOG'S AGE Incorrect
Laya PET DOG'S AGE Incorrect
CalDec v1 SECOND SISTER'S AGE Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "8369",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "d4f2b5b8635d6d09577a245da1eb96a93ab865ecfa2f43ffedb4e0a936dad6d3"
}

Type to search questions, variants and documentation.