RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 47

Back to Information seeking · All capabilities

Question ID: questbench/26e575472c7174cae9e3
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Carlos, Jim, and Carrey were at the beach playing and they decided to gather some seashells. Jim collected 27 seashells, which was more than what Carlos collected. Carlos collected twice as many as Carrey. They gathered all their seashells and divided them equally between themselves. How many did each person get?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • F1: The difference between Jim and Carlos' seashells
  • X: The number of seashells the seagull stole and a friendly beachcomber gave back to Jim

Reference answer: The difference between Jim and Carlos' seashells (F1)

Exact evaluated input as JSON
{
  "state": "Carlos, Jim, and Carrey were at the beach playing and they decided to gather some seashells. Jim collected 27 seashells, which was more than what Carlos collected. Carlos collected twice as many as Carrey. They gathered all their seashells and divided them equally between themselves. How many did each person get?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "F1": "The difference between Jim and Carlos' seashells",
    "X": "The number of seashells the seagull stole and a friendly beachcomber gave back to Jim"
  }
}

Recorded model answers

Model Answer Outcome
Astra The difference between Jim and Carlos' seashells Correct
Opus 5.5 The difference between Jim and Carlos' seashells Correct
GLiDE v1.2 The difference between Jim and Carlos' seashells Correct
Decider v1.1 The difference between Jim and Carlos' seashells Correct
Jev The difference between Jim and Carlos' seashells Correct
Microsoft Decision-1 The difference between Jim and Carlos' seashells Correct
Clef The difference between Jim and Carlos' seashells Correct
Luna Decisions The difference between Jim and Carlos' seashells Correct
Clef Flash The difference between Jim and Carlos' seashells Correct
Strands Decider The difference between Jim and Carlos' seashells Correct
GLiNER2.5 Decide The difference between Jim and Carlos' seashells Correct
Laya The difference between Jim and Carlos' seashells Correct
CalDec v1 The difference between Jim and Carlos' seashells Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "6883",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "4b2615c64f5cc77c9ae0c3d79209e01477468869866a7fc2bf11f99b1ee89c82"
}

Type to search questions, variants and documentation.