Social cognition · question 55
Back to Social cognition · All capabilities
Question ID: bigtom/81b37eb7a2b556b39d96
Source: BigToM
Input format: native choices
Explore this question’s altered versions
Context
Lila is a talented violinist who lives in a bustling city in India. Today, she has an important audition for a renowned symphony orchestra. Lila picks up her violin case, which she assumes contains her valuable violin. Lila believes that her violin is in the case. However, her younger brother, who was playing around earlier, accidentally swapped her violin case with his identical-looking case containing a beginner's violin. Lila does not notice the swapped violin cases.
Question
Does Lila believe the violin case contains her valuable violin or a beginner's violin?
Answer choices
- A: Lila believes the violin case contains a beginner's violin.
- B: Lila believes the violin case contains her valuable violin.
Reference answer: Lila believes the violin case contains her valuable violin. (B)
Exact evaluated input as JSON
{
"state": "Lila is a talented violinist who lives in a bustling city in India. Today, she has an important audition for a renowned symphony orchestra. Lila picks up her violin case, which she assumes contains her valuable violin. Lila believes that her violin is in the case. However, her younger brother, who was playing around earlier, accidentally swapped her violin case with his identical-looking case containing a beginner's violin. Lila does not notice the swapped violin cases.",
"question": "Does Lila believe the violin case contains her valuable violin or a beginner's violin?",
"choices": {
"A": "Lila believes the violin case contains a beginner's violin.",
"B": "Lila believes the violin case contains her valuable violin."
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Lila believes the violin case contains her valuable violin. | Correct |
| Opus 5.5 | Lila believes the violin case contains her valuable violin. | Correct |
| GLiDE v1.2 | Lila believes the violin case contains her valuable violin. | Correct |
| Decider v1.1 | Lila believes the violin case contains her valuable violin. | Correct |
| Jev | Lila believes the violin case contains her valuable violin. | Correct |
| Microsoft Decision-1 | Lila believes the violin case contains her valuable violin. | Correct |
| Clef | Lila believes the violin case contains her valuable violin. | Correct |
| Luna Decisions | Lila believes the violin case contains her valuable violin. | Correct |
| Clef Flash | Lila believes the violin case contains her valuable violin. | Correct |
| Strands Decider | Lila believes the violin case contains her valuable violin. | Correct |
| GLiNER2.5 Decide | Lila believes the violin case contains her valuable violin. | Correct |
| Laya | Lila believes the violin case contains her valuable violin. | Correct |
| CalDec v1 | Lila believes the violin case contains a beginner's violin. | Incorrect |
Source and sharing
Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/bigtom/bigtom.csv",
"record": "94/forward-belief/false",
"kind": "source_file_record"
},
"source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
"input_hash": "e63fde8e0924dbd836ecdaa497d762273a3824eb63940b3c6f13d477a0efe472"
}