Social cognition · question 8
Back to Social cognition · All capabilities
Question ID: bigtom/84e0b56db0941d7c5017
Source: BigToM
Input format: native choices
Explore this question’s altered versions
Context
Zara is a skilled tailor in a small Moroccan town. She is working on a custom-made dress for a client's wedding. Zara notices that the silk fabric she chose for the dress is of high quality, smooth and shiny. Zara believes that the silk fabric is indeed high quality, smooth, and shiny. While Zara is taking a break, her assistant accidentally spills a cup of coffee on the silk fabric, staining and damaging it. Zara does not notice the coffee spill on the silk fabric.
Question
Does Zara believe the silk fabric is high quality, smooth, and shiny, or stained and damaged?
Answer choices
- A: Zara believes the silk fabric is stained and damaged.
- B: Zara believes the silk fabric is high quality, smooth, and shiny.
Reference answer: Zara believes the silk fabric is high quality, smooth, and shiny. (B)
Exact evaluated input as JSON
{
"state": "Zara is a skilled tailor in a small Moroccan town. She is working on a custom-made dress for a client's wedding. Zara notices that the silk fabric she chose for the dress is of high quality, smooth and shiny. Zara believes that the silk fabric is indeed high quality, smooth, and shiny. While Zara is taking a break, her assistant accidentally spills a cup of coffee on the silk fabric, staining and damaging it. Zara does not notice the coffee spill on the silk fabric.",
"question": "Does Zara believe the silk fabric is high quality, smooth, and shiny, or stained and damaged?",
"choices": {
"A": "Zara believes the silk fabric is stained and damaged.",
"B": "Zara believes the silk fabric is high quality, smooth, and shiny."
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Opus 5.5 | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| GLiDE v1.2 | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Decider v1.1 | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Jev | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Microsoft Decision-1 | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Clef | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Luna Decisions | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Clef Flash | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Strands Decider | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| GLiNER2.5 Decide | Zara believes the silk fabric is high quality, smooth, and shiny. | Correct |
| Laya | Zara believes the silk fabric is stained and damaged. | Incorrect |
| CalDec v1 | Zara believes the silk fabric is stained and damaged. | Incorrect |
Source and sharing
Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/bigtom/bigtom.csv",
"record": "182/forward-belief/false",
"kind": "source_file_record"
},
"source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
"input_hash": "f7e74a5c4d022968455da12d47cb0618cdb1ead8b5b2cce718f632fd988df9d6"
}