RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 15

Back to Social cognition · All capabilities

Question ID: bigtom/6923220e404f5a8118af
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Ulysses is a zookeeper in a large city, responsible for taking care of a rare albino peacock. He wants to make sure the peacock is safely inside its enclosure before the zoo opens for the day. He sees that the peacock is perched on a branch inside the enclosure. Ulysses believes the peacock is inside the enclosure. A sudden gust of wind knocks the peacock off its perch, causing it to fly out of the enclosure and land in a nearby tree. Ulysses does not see the peacock in the tree.

Question

Does Ulysses believe the albino peacock is inside the enclosure or outside in a nearby tree?

Answer choices

  • A: Ulysses believes the albino peacock is outside in a nearby tree.
  • B: Ulysses believes the albino peacock is inside the enclosure.

Reference answer: Ulysses believes the albino peacock is inside the enclosure. (B)

Exact evaluated input as JSON
{
  "state": "Ulysses is a zookeeper in a large city, responsible for taking care of a rare albino peacock. He wants to make sure the peacock is safely inside its enclosure before the zoo opens for the day. He sees that the peacock is perched on a branch inside the enclosure. Ulysses believes the peacock is inside the enclosure. A sudden gust of wind knocks the peacock off its perch, causing it to fly out of the enclosure and land in a nearby tree. Ulysses does not see the peacock in the tree.",
  "question": "Does Ulysses believe the albino peacock is inside the enclosure or outside in a nearby tree?",
  "choices": {
    "A": "Ulysses believes the albino peacock is outside in a nearby tree.",
    "B": "Ulysses believes the albino peacock is inside the enclosure."
  }
}

Recorded model answers

Model Answer Outcome
Astra Ulysses believes the albino peacock is inside the enclosure. Correct
Opus 5.5 Ulysses believes the albino peacock is inside the enclosure. Correct
GLiDE v1.2 Ulysses believes the albino peacock is inside the enclosure. Correct
Decider v1.1 Ulysses believes the albino peacock is inside the enclosure. Correct
Jev Ulysses believes the albino peacock is inside the enclosure. Correct
Microsoft Decision-1 Ulysses believes the albino peacock is inside the enclosure. Correct
Clef Ulysses believes the albino peacock is inside the enclosure. Correct
Luna Decisions Ulysses believes the albino peacock is inside the enclosure. Correct
Clef Flash Ulysses believes the albino peacock is inside the enclosure. Correct
Strands Decider Ulysses believes the albino peacock is outside in a nearby tree. Incorrect
GLiNER2.5 Decide Ulysses believes the albino peacock is inside the enclosure. Correct
Laya Ulysses believes the albino peacock is inside the enclosure. Correct
CalDec v1 Ulysses believes the albino peacock is outside in a nearby tree. Incorrect

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "128/forward-belief/false",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "dbb568572f3b2f84e86b0533c6e873de7902a098b346e653ab73e43c5e07829e"
}

Type to search questions, variants and documentation.