RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 21

Back to Information seeking · All capabilities

Question ID: questbench/d888f63b8d3a55bb40f5
Source: QuestBench (Google DeepMind)
Input format: native choices

Explore this question’s altered versions

Context

Jackie is trying to decide whether to do her taxes herself or hire an accountant. If she does the taxes herself, she'll be able to do 3 fewer hours of freelance work. The accountant charges $90. However, she also has a side hobby of painting. She has paintings ready to sell, but this will not affect her decision about the taxes. How much more money will she have if she hires the accountant?

Question

Which missing variable should be asked about to solve this problem?

Answer choices

  • B: freelance rate per hour
  • D: per painting cost
  • E: number of paintings

Reference answer: freelance rate per hour (B)

Exact evaluated input as JSON
{
  "state": "Jackie is trying to decide whether to do her taxes herself or hire an accountant. If she does the taxes herself, she'll be able to do 3 fewer hours of freelance work. The accountant charges $90. However, she also has a side hobby of painting. She has paintings ready to sell, but this will not affect her decision about the taxes. How much more money will she have if she hires the accountant?",
  "question": "Which missing variable should be asked about to solve this problem?",
  "choices": {
    "B": "freelance rate per hour",
    "D": "per painting cost",
    "E": "number of paintings"
  }
}

Recorded model answers

Model Answer Outcome
Astra freelance rate per hour Correct
Opus 5.5 freelance rate per hour Correct
GLiDE v1.2 freelance rate per hour Correct
Decider v1.1 freelance rate per hour Correct
Jev freelance rate per hour Correct
Microsoft Decision-1 freelance rate per hour Correct
Clef freelance rate per hour Correct
Luna Decisions freelance rate per hour Correct
Clef Flash freelance rate per hour Correct
Strands Decider freelance rate per hour Correct
GLiNER2.5 Decide freelance rate per hour Correct
Laya freelance rate per hour Correct
CalDec v1 freelance rate per hour Correct

Source and sharing

Attribution: Google LLC; QuestBench authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/questbench.tar.gz/questbench_data/GSM-Q/gsm_verbal_full.csv",
    "record": "16498",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "6aa7f970cf89904350075f34a53c1dc37e7efd82cced09c3f70afd9a3248e197"
}

Type to search questions, variants and documentation.