RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 65

Back to Social cognition · All capabilities

Question ID: tombench/30f4e24516102c7145bc
Source: ToMBench
Input format: native choices

Explore this question’s altered versions

Context

Xiao Ling promised to help her mom buy groceries. Her mom told her to make sure to buy carrots. When she left, she ran into her good friend, and they went to buy groceries together. Xiao Ling bought a lot of groceries and, in the joy of chatting, forgot to buy carrots. Her mom asks, "Have you brought back all the groceries?" Xiao Ling looks at the many groceries and thinks for a moment and says, "I've brought them back."

Question

Why does Xiaoling say this?

Answer choices

  • A: Xiaoling deliberately lies because she does not want her mother to be angry.
  • B: Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend.
  • C: Xiaoling deliberately lies because she wants more free time with her friends.
  • D: Xiaoling thinks she buys all the groceries because she is usually very careful.

Reference answer: Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. (B)

Exact evaluated input as JSON
{
  "state": "Xiao Ling promised to help her mom buy groceries. Her mom told her to make sure to buy carrots. When she left, she ran into her good friend, and they went to buy groceries together. Xiao Ling bought a lot of groceries and, in the joy of chatting, forgot to buy carrots. Her mom asks, \"Have you brought back all the groceries?\" Xiao Ling looks at the many groceries and thinks for a moment and says, \"I've brought them back.\"",
  "question": "Why does Xiaoling say this?",
  "choices": {
    "A": "Xiaoling deliberately lies because she does not want her mother to be angry.",
    "B": "Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend.",
    "C": "Xiaoling deliberately lies because she wants more free time with her friends.",
    "D": "Xiaoling thinks she buys all the groceries because she is usually very careful."
  }
}

Recorded model answers

Model Answer Outcome
Astra Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Opus 5.5 Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
GLiDE v1.2 Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Decider v1.1 Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Jev Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Microsoft Decision-1 Xiaoling deliberately lies because she does not want her mother to be angry. Incorrect
Clef Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Luna Decisions Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Clef Flash Xiaoling deliberately lies because she does not want her mother to be angry. Incorrect
Strands Decider Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
GLiNER2.5 Decide Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
Laya Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct
CalDec v1 Xiaoling forgets that she does not buy carrots because she is too happy chatting with her friend. Correct

Source and sharing

Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/Strange Story Task.jsonl",
    "record": "47",
    "kind": "source_file_record"
  },
  "source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
  "input_hash": "cc66c2ca9df6ada8d7515f53774aff567f1ce231e7f5c921eef558a75a123473"
}

Type to search questions, variants and documentation.