RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 9

Back to Information seeking · All capabilities

Question ID: questbench/a347ec5f79aad695fc48
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

At a wedding Dylan attended, there were guests in the reception. Each guest brought a plate of 15 cookies, and there were also 10 waiters who each brought cookies. However, the waiters' cookies were not included in the bride's decision to give away cookies. The bride decided to give 1/2 of the guests' cookies to the church next door as a thank you for helping in the wedding reception. If each person in the church next door got 15 cookies, and the church also received an additional cookies from a local bakery which were not distributed, how many people were in the church next door?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • A: Number of guests' cookies
  • D: Number of waiters
  • E: Number of cookies in each waiter plates
  • J: Number of cookies from local bakery

Reference answer: Number of guests' cookies (A)

Exact evaluated input as JSON
{
  "state": "At a wedding Dylan attended, there were guests in the reception. Each guest brought a plate of 15 cookies, and there were also 10 waiters who each brought cookies. However, the waiters' cookies were not included in the bride's decision to give away cookies. The bride decided to give 1/2 of the guests' cookies to the church next door as a thank you for helping in the wedding reception. If each person in the church next door got 15 cookies, and the church also received an additional cookies from a local bakery which were not distributed, how many people were in the church next door?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "A": "Number of guests' cookies",
    "D": "Number of waiters",
    "E": "Number of cookies in each waiter plates",
    "J": "Number of cookies from local bakery"
  }
}

Recorded model answers

Model Answer Outcome
Astra Number of guests' cookies Correct
Opus 5.5 Number of guests' cookies Correct
GLiDE v1.2 Number of guests' cookies Correct
Decider v1.1 Number of guests' cookies Correct
Jev Number of cookies from local bakery Incorrect
Microsoft Decision-1 Number of cookies from local bakery Incorrect
Clef Number of guests' cookies Correct
Luna Decisions Number of guests' cookies Correct
Clef Flash Number of cookies from local bakery Incorrect
Strands Decider Number of cookies from local bakery Incorrect
GLiNER2.5 Decide Number of cookies from local bakery Incorrect
Laya Number of guests' cookies Correct
CalDec v1 Number of guests' cookies Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "2059",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "aaa93072fb82ab47078cc4eac6148eb0bab0163c59a650a85b9707db26f17e43"
}

Type to search questions, variants and documentation.