Retrieval · question 4
Back to Retrieval · All capabilities
Question ID: tombench/1330d8ae29b9eb49cbc0
Source: ToMBench
Input format: native choices
Explore this question’s altered versions
Context
Xiao Bei finds a suitcase in the parking lot, the label on the suitcase is corn, Xiao Bei cannot see what is inside the suitcase, Xiao Bei opens the suitcase and finds a pencil, there is no corn inside the suitcase, Xiao Bei closes the suitcase and puts it back in its place, Xiao Ming enters the parking lot and sees the suitcase.
Question
What is inside the suitcase?
Answer choices
- A: Pencil
- B: Compass
- C: Corn
- D: Watch
Reference answer: Pencil (A)
Exact evaluated input as JSON
{
"state": "Xiao Bei finds a suitcase in the parking lot, the label on the suitcase is corn, Xiao Bei cannot see what is inside the suitcase, Xiao Bei opens the suitcase and finds a pencil, there is no corn inside the suitcase, Xiao Bei closes the suitcase and puts it back in its place, Xiao Ming enters the parking lot and sees the suitcase.",
"question": "What is inside the suitcase?",
"choices": {
"A": "Pencil",
"B": "Compass",
"C": "Corn",
"D": "Watch"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Pencil | Correct |
| Opus 5.5 | Pencil | Correct |
| GLiDE v1.2 | Pencil | Correct |
| Decider v1.1 | Pencil | Correct |
| Jev | Pencil | Correct |
| Microsoft Decision-1 | Pencil | Correct |
| Clef | Pencil | Correct |
| Luna Decisions | Pencil | Correct |
| Clef Flash | Pencil | Correct |
| Strands Decider | Pencil | Correct |
| GLiNER2.5 Decide | Corn | Incorrect |
| Laya | Pencil | Correct |
| CalDec v1 | Pencil | Correct |
Source and sharing
Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/False Belief Task.jsonl",
"record": "420",
"kind": "source_file_record"
},
"source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
"input_hash": "fdbea2453685f95bbedc42a3266c49dfbdce1684f5a398d12ed0f57f08642f1a"
}