RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 51

Back to Social cognition · All capabilities

Question ID: bigtom/a8a9ca945741c67cdecb
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Julius is at a community garden, tending to his vegetable plot. Julius wants to harvest some ripe tomatoes for dinner tonight. The tomatoes on the vine appear to be plump and ripe to Julius, with a vibrant red color. Julius believes that the tomatoes are ripe and ready to be picked. A sudden hailstorm strikes the garden, damaging the tomatoes and leaving them bruised and battered. Julius notices the hailstorm.

Question

Does Julius believe the tomatoes are ripe and ready to be picked or bruised and battered?

Answer choices

  • A: Julius believes the tomatoes are bruised and battered.
  • B: Julius believes the tomatoes are ripe and ready to be picked.

Reference answer: Julius believes the tomatoes are bruised and battered. (A)

Exact evaluated input as JSON
{
  "state": "Julius is at a community garden, tending to his vegetable plot. Julius wants to harvest some ripe tomatoes for dinner tonight. The tomatoes on the vine appear to be plump and ripe to Julius, with a vibrant red color. Julius believes that the tomatoes are ripe and ready to be picked. A sudden hailstorm strikes the garden, damaging the tomatoes and leaving them bruised and battered. Julius notices the hailstorm.",
  "question": "Does Julius believe the tomatoes are ripe and ready to be picked or bruised and battered?",
  "choices": {
    "A": "Julius believes the tomatoes are bruised and battered.",
    "B": "Julius believes the tomatoes are ripe and ready to be picked."
  }
}

Recorded model answers

Model Answer Outcome
Astra Julius believes the tomatoes are bruised and battered. Correct
Opus 5.5 Julius believes the tomatoes are bruised and battered. Correct
GLiDE v1.2 Julius believes the tomatoes are bruised and battered. Correct
Decider v1.1 Julius believes the tomatoes are bruised and battered. Correct
Jev Julius believes the tomatoes are ripe and ready to be picked. Incorrect
Microsoft Decision-1 Julius believes the tomatoes are ripe and ready to be picked. Incorrect
Clef Julius believes the tomatoes are bruised and battered. Correct
Luna Decisions Julius believes the tomatoes are ripe and ready to be picked. Incorrect
Clef Flash Julius believes the tomatoes are bruised and battered. Correct
Strands Decider Julius believes the tomatoes are bruised and battered. Correct
GLiNER2.5 Decide Julius believes the tomatoes are ripe and ready to be picked. Incorrect
Laya Julius believes the tomatoes are bruised and battered. Correct
CalDec v1 Julius believes the tomatoes are bruised and battered. Correct

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "152/forward-belief/true",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "12fdbc0e2a32483ee8f4b379e764d0ba7c5fcb4493601b7fcc2036dffcd5c268"
}

Type to search questions, variants and documentation.