Social cognition · question 12
Back to Social cognition · All capabilities
Question ID: bigtom/4d949574e47d4f33c824
Source: BigToM
Input format: native choices
Explore this question’s altered versions
Context
Delia is a scientist working in a remote research station in Antarctica. She needs to collect important data from a weather monitoring device outside the station. Dalia sees that the device is displaying the correct data. Delia believes the device is displaying the correct data. Suddenly, a powerful solar flare disrupts the device's sensors, causing it to display incorrect data. Delia does not observe the solar flare.
Question
Does Delia believe the weather monitoring device is displaying correct or incorrect data?
Answer choices
- A: Delia believes the weather monitoring device is displaying incorrect data.
- B: Delia believes the weather monitoring device is displaying correct data.
Reference answer: Delia believes the weather monitoring device is displaying correct data. (B)
Exact evaluated input as JSON
{
"state": "Delia is a scientist working in a remote research station in Antarctica. She needs to collect important data from a weather monitoring device outside the station. Dalia sees that the device is displaying the correct data. Delia believes the device is displaying the correct data. Suddenly, a powerful solar flare disrupts the device's sensors, causing it to display incorrect data. Delia does not observe the solar flare.",
"question": "Does Delia believe the weather monitoring device is displaying correct or incorrect data?",
"choices": {
"A": "Delia believes the weather monitoring device is displaying incorrect data.",
"B": "Delia believes the weather monitoring device is displaying correct data."
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Opus 5.5 | Delia believes the weather monitoring device is displaying correct data. | Correct |
| GLiDE v1.2 | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Decider v1.1 | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Jev | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Microsoft Decision-1 | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Clef | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Luna Decisions | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Clef Flash | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Strands Decider | Delia believes the weather monitoring device is displaying correct data. | Correct |
| GLiNER2.5 Decide | Delia believes the weather monitoring device is displaying correct data. | Correct |
| Laya | Delia believes the weather monitoring device is displaying incorrect data. | Incorrect |
| CalDec v1 | Delia believes the weather monitoring device is displaying incorrect data. | Incorrect |
Source and sharing
Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "data/bigtom/bigtom.csv",
"record": "140/forward-belief/false",
"kind": "source_file_record"
},
"source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
"input_hash": "14f184fb2b04a2431c9b0b7d732e57d7828202997cfc55ab0f90f8ac93b843a0"
}