Social cognition · question 17
Back to Social cognition · All capabilities
Question ID: tombench/22239a3652a340bfad6e
Source: ToMBench
Input format: native choices
Explore this question’s altered versions
Context
Xiao Bei and Xiao Li are wandering in the parking lot, they see boxes and baskets, they find bananas in the box, Xiao Li leaves the parking lot, Xiao Bei moves the bananas to the basket.
Question
After Xiao Li returns to the parking lot, where does Xiao Bei think Xiao Li looks for the bananas?
Answer choices
- A: Briefcase
- B: Box
- C: Basket
- D: Cabinet
Reference answer: Box (B)
Exact evaluated input as JSON
{
"state": "Xiao Bei and Xiao Li are wandering in the parking lot, they see boxes and baskets, they find bananas in the box, Xiao Li leaves the parking lot, Xiao Bei moves the bananas to the basket.",
"question": "After Xiao Li returns to the parking lot, where does Xiao Bei think Xiao Li looks for the bananas?",
"choices": {
"A": "Briefcase",
"B": "Box",
"C": "Basket",
"D": "Cabinet"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Box | Correct |
| Opus 5.5 | Box | Correct |
| GLiDE v1.2 | Box | Correct |
| Decider v1.1 | Box | Correct |
| Jev | Box | Correct |
| Microsoft Decision-1 | Box | Correct |
| Clef | Box | Correct |
| Luna Decisions | Box | Correct |
| Clef Flash | Box | Correct |
| Strands Decider | Box | Correct |
| GLiNER2.5 Decide | Basket | Incorrect |
| Laya | Box | Correct |
| CalDec v1 | Box | Correct |
Source and sharing
Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/False Belief Task.jsonl",
"record": "286",
"kind": "source_file_record"
},
"source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
"input_hash": "99e9b90aec512e5e5fb92da234eb4ab2c57182747d1a45b9e94439dc5ece039c"
}