Communication · question 23
Back to Communication · All capabilities
Question ID: tombench/f6e207cefe0db85a1890
Source: ToMBench
Input format: native choices
Explore this question’s altered versions
Context
In the national library, Xiao Hua is reading a book seriously. Then a girl comes in, sits next to Xiao Hua, she turns on the music player on her phone, the volume is very loud. Xiao Hua lifts his head and looks at her, then continues to read his book and says, "Excuse me, is this the national library?"
Question
What does Xiao Hua really want to say when he says this?
Answer choices
- A: Xiao Hua is asking about his current location
- B: Xiao Hua wants the girl to read books with him.
- C: Xiao Hua is interested in the music on the girl's phone.
- D: Xiao Hua hints the girl to turn off the music on her phone.
Reference answer: Xiao Hua hints the girl to turn off the music on her phone. (D)
Exact evaluated input as JSON
{
"state": "In the national library, Xiao Hua is reading a book seriously. Then a girl comes in, sits next to Xiao Hua, she turns on the music player on her phone, the volume is very loud. Xiao Hua lifts his head and looks at her, then continues to read his book and says, \"Excuse me, is this the national library?\"",
"question": "What does Xiao Hua really want to say when he says this?",
"choices": {
"A": "Xiao Hua is asking about his current location",
"B": "Xiao Hua wants the girl to read books with him.",
"C": "Xiao Hua is interested in the music on the girl's phone.",
"D": "Xiao Hua hints the girl to turn off the music on her phone."
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Opus 5.5 | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| GLiDE v1.2 | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Decider v1.1 | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Jev | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Microsoft Decision-1 | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Clef | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Luna Decisions | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Clef Flash | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| Strands Decider | Xiao Hua hints the girl to turn off the music on her phone. | Correct |
| GLiNER2.5 Decide | Xiao Hua is asking about his current location | Incorrect |
| Laya | Xiao Hua is asking about his current location | Incorrect |
| CalDec v1 | Xiao Hua is asking about his current location | Incorrect |
Source and sharing
Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/Hinting Task Test.jsonl",
"record": "66",
"kind": "source_file_record"
},
"source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
"input_hash": "931eb0561cbe10f4f7d9bfae50acdf51a3b47e4c680893fd515c0e9f00153736"
}