RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 55

Back to Social cognition · All capabilities

Question ID: bigtom/81b37eb7a2b556b39d96
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Lila is a talented violinist who lives in a bustling city in India. Today, she has an important audition for a renowned symphony orchestra. Lila picks up her violin case, which she assumes contains her valuable violin. Lila believes that her violin is in the case. However, her younger brother, who was playing around earlier, accidentally swapped her violin case with his identical-looking case containing a beginner's violin. Lila does not notice the swapped violin cases.

Question

Does Lila believe the violin case contains her valuable violin or a beginner's violin?

Answer choices

  • A: Lila believes the violin case contains a beginner's violin.
  • B: Lila believes the violin case contains her valuable violin.

Reference answer: Lila believes the violin case contains her valuable violin. (B)

Exact evaluated input as JSON
{
  "state": "Lila is a talented violinist who lives in a bustling city in India. Today, she has an important audition for a renowned symphony orchestra. Lila picks up her violin case, which she assumes contains her valuable violin. Lila believes that her violin is in the case. However, her younger brother, who was playing around earlier, accidentally swapped her violin case with his identical-looking case containing a beginner's violin. Lila does not notice the swapped violin cases.",
  "question": "Does Lila believe the violin case contains her valuable violin or a beginner's violin?",
  "choices": {
    "A": "Lila believes the violin case contains a beginner's violin.",
    "B": "Lila believes the violin case contains her valuable violin."
  }
}

Recorded model answers

Model Answer Outcome
Astra Lila believes the violin case contains her valuable violin. Correct
Opus 5.5 Lila believes the violin case contains her valuable violin. Correct
GLiDE v1.2 Lila believes the violin case contains her valuable violin. Correct
Decider v1.1 Lila believes the violin case contains her valuable violin. Correct
Jev Lila believes the violin case contains her valuable violin. Correct
Microsoft Decision-1 Lila believes the violin case contains her valuable violin. Correct
Clef Lila believes the violin case contains her valuable violin. Correct
Luna Decisions Lila believes the violin case contains her valuable violin. Correct
Clef Flash Lila believes the violin case contains her valuable violin. Correct
Strands Decider Lila believes the violin case contains her valuable violin. Correct
GLiNER2.5 Decide Lila believes the violin case contains her valuable violin. Correct
Laya Lila believes the violin case contains her valuable violin. Correct
CalDec v1 Lila believes the violin case contains a beginner's violin. Incorrect

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "94/forward-belief/false",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "e63fde8e0924dbd836ecdaa497d762273a3824eb63940b3c6f13d477a0efe472"
}

Type to search questions, variants and documentation.