State tracking and updating · question 3
Back to State tracking and updating · All capabilities
Question ID: tombench/f081edc079ce9161fefc
Source: ToMBench
Input format: native choices
Explore this question’s altered versions
Context
Xiao Ming and Zhang Wei are wandering in the hall, they see a handbag and a box, they find a sponge in the handbag, Zhang Wei leaves the hall, Xiao Ming moves the sponge to the box.
Question
Where is the sponge now?
Answer choices
- A: Box
- B: Handbag
- C: Storage locker
- D: Briefcase
Reference answer: Box (A)
Exact evaluated input as JSON
{
"state": "Xiao Ming and Zhang Wei are wandering in the hall, they see a handbag and a box, they find a sponge in the handbag, Zhang Wei leaves the hall, Xiao Ming moves the sponge to the box.",
"question": "Where is the sponge now?",
"choices": {
"A": "Box",
"B": "Handbag",
"C": "Storage locker",
"D": "Briefcase"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Box | Correct |
| Opus 5.5 | Box | Correct |
| GLiDE v1.2 | Box | Correct |
| Decider v1.1 | Box | Correct |
| Jev | Box | Correct |
| Microsoft Decision-1 | Box | Correct |
| Clef | Box | Correct |
| Luna Decisions | Box | Correct |
| Clef Flash | Box | Correct |
| Strands Decider | Box | Correct |
| GLiNER2.5 Decide | Box | Correct |
| Laya | Box | Correct |
| CalDec v1 | Box | Correct |
Source and sharing
Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/False Belief Task.jsonl",
"record": "270",
"kind": "source_file_record"
},
"source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
"input_hash": "0dde839eb1142c00462a69f9915638eba8e5f4d9c35c92666b6b4d8304c51eb6"
}