RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 4

Back to Retrieval · All capabilities

Question ID: tombench/1330d8ae29b9eb49cbc0
Source: ToMBench
Input format: native choices

Explore this question’s altered versions

Context

Xiao Bei finds a suitcase in the parking lot, the label on the suitcase is corn, Xiao Bei cannot see what is inside the suitcase, Xiao Bei opens the suitcase and finds a pencil, there is no corn inside the suitcase, Xiao Bei closes the suitcase and puts it back in its place, Xiao Ming enters the parking lot and sees the suitcase.

Question

What is inside the suitcase?

Answer choices

  • A: Pencil
  • B: Compass
  • C: Corn
  • D: Watch

Reference answer: Pencil (A)

Exact evaluated input as JSON
{
  "state": "Xiao Bei finds a suitcase in the parking lot, the label on the suitcase is corn, Xiao Bei cannot see what is inside the suitcase, Xiao Bei opens the suitcase and finds a pencil, there is no corn inside the suitcase, Xiao Bei closes the suitcase and puts it back in its place, Xiao Ming enters the parking lot and sees the suitcase.",
  "question": "What is inside the suitcase?",
  "choices": {
    "A": "Pencil",
    "B": "Compass",
    "C": "Corn",
    "D": "Watch"
  }
}

Recorded model answers

Model Answer Outcome
Astra Pencil Correct
Opus 5.5 Pencil Correct
GLiDE v1.2 Pencil Correct
Decider v1.1 Pencil Correct
Jev Pencil Correct
Microsoft Decision-1 Pencil Correct
Clef Pencil Correct
Luna Decisions Pencil Correct
Clef Flash Pencil Correct
Strands Decider Pencil Correct
GLiNER2.5 Decide Corn Incorrect
Laya Pencil Correct
CalDec v1 Pencil Correct

Source and sharing

Attribution: Zhuang Chen et al.; ToMBench authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/False Belief Task.jsonl",
    "record": "420",
    "kind": "source_file_record"
  },
  "source_revision": "4f491f0784ed31ef93a8837615fbc48f885cf78d",
  "input_hash": "fdbea2453685f95bbedc42a3266c49dfbdce1684f5a398d12ed0f57f08642f1a"
}

Type to search questions, variants and documentation.