RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 47

Back to Social cognition · All capabilities

Question ID: bigtom/fc62e666199d3ebd1c45
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Leilani is a professional chef preparing for a cooking competition. She needs to make a perfect chocolate soufflé to impress the judges. Leilani observes that her oven is preheated to the precise temperature required for baking the soufflé. Leilani believes that the oven is at the perfect temperature for her soufflé. However, a power outage occurs, causing the oven to lose heat rapidly. Leilani notices the power outage and the oven turning off.

Question

Does Leilani believe the oven is at the perfect temperature or that it has lost heat due to the power outage?

Answer choices

  • A: Leilani believes the oven has lost heat due to the power outage.
  • B: Leilani believes the oven is at the perfect temperature.

Reference answer: Leilani believes the oven has lost heat due to the power outage. (A)

Exact evaluated input as JSON
{
  "state": "Leilani is a professional chef preparing for a cooking competition. She needs to make a perfect chocolate soufflé to impress the judges. Leilani observes that her oven is preheated to the precise temperature required for baking the soufflé. Leilani believes that the oven is at the perfect temperature for her soufflé. However, a power outage occurs, causing the oven to lose heat rapidly. Leilani notices the power outage and the oven turning off.",
  "question": "Does Leilani believe the oven is at the perfect temperature or that it has lost heat due to the power outage?",
  "choices": {
    "A": "Leilani believes the oven has lost heat due to the power outage.",
    "B": "Leilani believes the oven is at the perfect temperature."
  }
}

Recorded model answers

Model Answer Outcome
Astra Leilani believes the oven has lost heat due to the power outage. Correct
Opus 5.5 Leilani believes the oven has lost heat due to the power outage. Correct
GLiDE v1.2 Leilani believes the oven has lost heat due to the power outage. Correct
Decider v1.1 Leilani believes the oven has lost heat due to the power outage. Correct
Jev Leilani believes the oven has lost heat due to the power outage. Correct
Microsoft Decision-1 Leilani believes the oven is at the perfect temperature. Incorrect
Clef Leilani believes the oven has lost heat due to the power outage. Correct
Luna Decisions Leilani believes the oven has lost heat due to the power outage. Correct
Clef Flash Leilani believes the oven has lost heat due to the power outage. Correct
Strands Decider Leilani believes the oven has lost heat due to the power outage. Correct
GLiNER2.5 Decide Leilani believes the oven has lost heat due to the power outage. Correct
Laya Leilani believes the oven is at the perfect temperature. Incorrect
CalDec v1 Leilani believes the oven has lost heat due to the power outage. Correct

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "183/forward-belief/true",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "e9fcc06ca40e970a76686b7cd2a36303ed3cbb0253ca55115e04ea7771299c07"
}

Type to search questions, variants and documentation.