RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 4

Back to Social cognition · All capabilities

Question ID: bigtom/b20aa68775f6a4d111e5
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Laila is walking through a beautiful park on a sunny day. Laila wants to take a picture of a stunning flowerbed she comes across. She sees that the flowers appear to be in full bloom, with vibrant colors. Laila believes the flowers are in full bloom. Suddenly, a heavy rainstorm hits the park, causing the flowers to wilt and lose their vibrancy. Laila does not notice the rainstorm's impact on the flowers.

Question

Does Laila believe the flowers are in full bloom or have wilted due to the rainstorm?

Answer choices

  • A: Laila believes the flowers have wilted due to the rainstorm.
  • B: Laila believes the flowers are in full bloom.

Reference answer: Laila believes the flowers are in full bloom. (B)

Exact evaluated input as JSON
{
  "state": "Laila is walking through a beautiful park on a sunny day. Laila wants to take a picture of a stunning flowerbed she comes across. She sees that the flowers appear to be in full bloom, with vibrant colors. Laila believes the flowers are in full bloom. Suddenly, a heavy rainstorm hits the park, causing the flowers to wilt and lose their vibrancy. Laila does not notice the rainstorm's impact on the flowers.",
  "question": "Does Laila believe the flowers are in full bloom or have wilted due to the rainstorm?",
  "choices": {
    "A": "Laila believes the flowers have wilted due to the rainstorm.",
    "B": "Laila believes the flowers are in full bloom."
  }
}

Recorded model answers

Model Answer Outcome
Astra Laila believes the flowers are in full bloom. Correct
Opus 5.5 Laila believes the flowers are in full bloom. Correct
GLiDE v1.2 Laila believes the flowers are in full bloom. Correct
Decider v1.1 Laila believes the flowers are in full bloom. Correct
Jev Laila believes the flowers are in full bloom. Correct
Microsoft Decision-1 Laila believes the flowers are in full bloom. Correct
Clef Laila believes the flowers are in full bloom. Correct
Luna Decisions Laila believes the flowers are in full bloom. Correct
Clef Flash Laila believes the flowers are in full bloom. Correct
Strands Decider Laila believes the flowers are in full bloom. Correct
GLiNER2.5 Decide Laila believes the flowers are in full bloom. Correct
Laya Laila believes the flowers are in full bloom. Correct
CalDec v1 Laila believes the flowers have wilted due to the rainstorm. Incorrect

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "122/forward-belief/false",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "436ccfd8b4abdd22a186ca82084a5d3c99416ec4030e0bad07b1d7f046f58f45"
}

Type to search questions, variants and documentation.