RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 55

Back to Information seeking · All capabilities

Question ID: questbench/02c9f7a5d303e4234dfd
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

James is putting together 4 tables. Each table has 4 legs and each leg needs screws. He is also putting chairs, which don't need screws. He has 40 screws. How many screws will he have left over?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • C: Number of screws required per leg.
  • E: Number of chairs James is assembling.

Reference answer: Number of screws required per leg. (C)

Exact evaluated input as JSON
{
  "state": "James is putting together 4 tables. Each table has 4 legs and each leg needs screws. He is also putting chairs, which don't need screws. He has 40 screws. How many screws will he have left over?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "C": "Number of screws required per leg.",
    "E": "Number of chairs James is assembling."
  }
}

Recorded model answers

Model Answer Outcome
Astra Number of screws required per leg. Correct
Opus 5.5 Number of screws required per leg. Correct
GLiDE v1.2 Number of screws required per leg. Correct
Decider v1.1 Number of screws required per leg. Correct
Jev Number of screws required per leg. Correct
Microsoft Decision-1 Number of screws required per leg. Correct
Clef Number of screws required per leg. Correct
Luna Decisions Number of screws required per leg. Correct
Clef Flash Number of screws required per leg. Correct
Strands Decider Number of chairs James is assembling. Incorrect
GLiNER2.5 Decide Number of screws required per leg. Correct
Laya Number of screws required per leg. Correct
CalDec v1 Number of screws required per leg. Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "10382",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "6994ba3df25a271a17d774baa4651e264da3d19fe642da7026ad38fa00dd183f"
}

Type to search questions, variants and documentation.