RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 39

Back to Information seeking · All capabilities

Question ID: questbench/38540b5a6f943e31294c
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Lindsay is doing the laundry, and thinks she has missed some socks. There are socks and pairs of trousers that need washing. If she washes 10 pairs of socks and 15 loose socks, how many socks has Lindsay missed?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • S: Total number of socks
  • X: Total pairs of trousers

Reference answer: Total number of socks (S)

Exact evaluated input as JSON
{
  "state": "Lindsay is doing the laundry, and thinks she has missed some socks. There are  socks and  pairs of trousers that need washing. If she washes 10 pairs of socks and 15 loose socks, how many socks has Lindsay missed?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "S": "Total number of socks",
    "X": "Total pairs of trousers"
  }
}

Recorded model answers

Model Answer Outcome
Astra Total number of socks Correct
Opus 5.5 Total number of socks Correct
GLiDE v1.2 Total number of socks Correct
Decider v1.1 Total number of socks Correct
Jev Total number of socks Correct
Microsoft Decision-1 Total number of socks Correct
Clef Total number of socks Correct
Luna Decisions Total number of socks Correct
Clef Flash Total number of socks Correct
Strands Decider Total number of socks Correct
GLiNER2.5 Decide Total number of socks Correct
Laya Total number of socks Correct
CalDec v1 Total number of socks Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "18176",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "d28fd5fa4f22be7486523a577c8cd7649ec5d756d8fa32f3657562a4b100909b"
}

Type to search questions, variants and documentation.