Information seeking · question 39
Back to Information seeking · All capabilities
Question ID: questbench/38540b5a6f943e31294c
Source: QuestBench (Google DeepMind)
Input format: native choices
Explore this question’s altered versions
Context
Lindsay is doing the laundry, and thinks she has missed some socks. There are socks and pairs of trousers that need washing. If she washes 10 pairs of socks and 15 loose socks, how many socks has Lindsay missed?
Question
Which missing variable should be asked about to solve this problem?
Answer choices
- S: Total number of socks
- X: Total pairs of trousers
Reference answer: Total number of socks (S)
Exact evaluated input as JSON
{
"state": "Lindsay is doing the laundry, and thinks she has missed some socks. There are socks and pairs of trousers that need washing. If she washes 10 pairs of socks and 15 loose socks, how many socks has Lindsay missed?",
"question": "Which missing variable should be asked about to solve this problem?",
"choices": {
"S": "Total number of socks",
"X": "Total pairs of trousers"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Total number of socks | Correct |
| Opus 5.5 | Total number of socks | Correct |
| GLiDE v1.2 | Total number of socks | Correct |
| Decider v1.1 | Total number of socks | Correct |
| Jev | Total number of socks | Correct |
| Microsoft Decision-1 | Total number of socks | Correct |
| Clef | Total number of socks | Correct |
| Luna Decisions | Total number of socks | Correct |
| Clef Flash | Total number of socks | Correct |
| Strands Decider | Total number of socks | Correct |
| GLiNER2.5 Decide | Total number of socks | Correct |
| Laya | Total number of socks | Correct |
| CalDec v1 | Total number of socks | Correct |
Source and sharing
Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
"record": "18176",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "d28fd5fa4f22be7486523a577c8cd7649ec5d756d8fa32f3657562a4b100909b"
}