RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 43

Back to Social cognition · All capabilities

Question ID: bigtom/b6b228f8546fb39a3dab
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Kofi is at the town market in search of fresh fruits and vegetables. Kofi wants to find a ripe watermelon to take home and share with his family. Kofi sees that the watermelon he picks up has a deep, hollow sound when he taps it. Kofi believes that the watermelon is ripe based on its sound. When Kofi is not looking, a small child accidentally drops the watermelon, causing it to crack and become damaged. Kofi notices the watermelon has been dropped and cracked.

Question

Does Kofi believe the watermelon is ripe and undamaged or cracked and damaged?

Answer choices

  • A: Kofi believes the watermelon is cracked and damaged.
  • B: Kofi believes the watermelon is ripe and undamaged.

Reference answer: Kofi believes the watermelon is cracked and damaged. (A)

Exact evaluated input as JSON
{
  "state": "Kofi is at the town market in search of fresh fruits and vegetables. Kofi wants to find a ripe watermelon to take home and share with his family. Kofi sees that the watermelon he picks up has a deep, hollow sound when he taps it. Kofi believes that the watermelon is ripe based on its sound. When Kofi is not looking, a small child accidentally drops the watermelon, causing it to crack and become damaged. Kofi notices the watermelon has been dropped and cracked.",
  "question": "Does Kofi believe the watermelon is ripe and undamaged or cracked and damaged?",
  "choices": {
    "A": "Kofi believes the watermelon is cracked and damaged.",
    "B": "Kofi believes the watermelon is ripe and undamaged."
  }
}

Recorded model answers

Model Answer Outcome
Astra Kofi believes the watermelon is cracked and damaged. Correct
Opus 5.5 Kofi believes the watermelon is cracked and damaged. Correct
GLiDE v1.2 Kofi believes the watermelon is cracked and damaged. Correct
Decider v1.1 Kofi believes the watermelon is cracked and damaged. Correct
Jev Kofi believes the watermelon is cracked and damaged. Correct
Microsoft Decision-1 Kofi believes the watermelon is cracked and damaged. Correct
Clef Kofi believes the watermelon is cracked and damaged. Correct
Luna Decisions Kofi believes the watermelon is cracked and damaged. Correct
Clef Flash Kofi believes the watermelon is cracked and damaged. Correct
Strands Decider Kofi believes the watermelon is cracked and damaged. Correct
GLiNER2.5 Decide Kofi believes the watermelon is cracked and damaged. Correct
Laya Kofi believes the watermelon is cracked and damaged. Correct
CalDec v1 Kofi believes the watermelon is cracked and damaged. Correct

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "103/forward-belief/true",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "b2d6fae7baa8322b70e41298b9ce96ceed861ee422f6dfad7bcff11e0ba5b052"
}

Type to search questions, variants and documentation.