Social cognition · question 43
Back to Social cognition · All capabilities
Question ID: bigtom/b6b228f8546fb39a3dab
Source: BigToM
Input format: native choices
Explore this question’s altered versions
Context
Kofi is at the town market in search of fresh fruits and vegetables. Kofi wants to find a ripe watermelon to take home and share with his family. Kofi sees that the watermelon he picks up has a deep, hollow sound when he taps it. Kofi believes that the watermelon is ripe based on its sound. When Kofi is not looking, a small child accidentally drops the watermelon, causing it to crack and become damaged. Kofi notices the watermelon has been dropped and cracked.
Question
Does Kofi believe the watermelon is ripe and undamaged or cracked and damaged?
Answer choices
- A: Kofi believes the watermelon is cracked and damaged.
- B: Kofi believes the watermelon is ripe and undamaged.
Reference answer: Kofi believes the watermelon is cracked and damaged. (A)
Exact evaluated input as JSON
{
"state": "Kofi is at the town market in search of fresh fruits and vegetables. Kofi wants to find a ripe watermelon to take home and share with his family. Kofi sees that the watermelon he picks up has a deep, hollow sound when he taps it. Kofi believes that the watermelon is ripe based on its sound. When Kofi is not looking, a small child accidentally drops the watermelon, causing it to crack and become damaged. Kofi notices the watermelon has been dropped and cracked.",
"question": "Does Kofi believe the watermelon is ripe and undamaged or cracked and damaged?",
"choices": {
"A": "Kofi believes the watermelon is cracked and damaged.",
"B": "Kofi believes the watermelon is ripe and undamaged."
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Kofi believes the watermelon is cracked and damaged. | Correct |
| Opus 5.5 | Kofi believes the watermelon is cracked and damaged. | Correct |
| GLiDE v1.2 | Kofi believes the watermelon is cracked and damaged. | Correct |
| Decider v1.1 | Kofi believes the watermelon is cracked and damaged. | Correct |
| Jev | Kofi believes the watermelon is cracked and damaged. | Correct |
| Microsoft Decision-1 | Kofi believes the watermelon is cracked and damaged. | Correct |
| Clef | Kofi believes the watermelon is cracked and damaged. | Correct |
| Luna Decisions | Kofi believes the watermelon is cracked and damaged. | Correct |
| Clef Flash | Kofi believes the watermelon is cracked and damaged. | Correct |
| Strands Decider | Kofi believes the watermelon is cracked and damaged. | Correct |
| GLiNER2.5 Decide | Kofi believes the watermelon is cracked and damaged. | Correct |
| Laya | Kofi believes the watermelon is cracked and damaged. | Correct |
| CalDec v1 | Kofi believes the watermelon is cracked and damaged. | Correct |
Source and sharing
Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/bigtom/bigtom.csv",
"record": "103/forward-belief/true",
"kind": "source_file_record"
},
"source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
"input_hash": "b2d6fae7baa8322b70e41298b9ce96ceed861ee422f6dfad7bcff11e0ba5b052"
}