State tracking and updating · question 50
Back to State tracking and updating · All capabilities
Question ID: bbh/a533319ac4eb062eb8b6
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a blue present, Bob has a red present, and Claire has a orange ball. As the event progresses, pairs of people swap gifts. First, Claire and Bob swap their gifts. Then, Alice and Claire swap their gifts. Finally, Bob and Alice swap their gifts. At the end of the event, Claire has the
Answer choices
- A: blue present
- B: red present
- C: orange ball
Reference answer: blue present (A)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a blue present, Bob has a red present, and Claire has a orange ball.\nAs the event progresses, pairs of people swap gifts. First, Claire and Bob swap their gifts. Then, Alice and Claire swap their gifts. Finally, Bob and Alice swap their gifts. At the end of the event, Claire has the",
"choices": {
"A": "blue present",
"B": "red present",
"C": "orange ball"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | blue present | Correct |
| Opus 5.5 | blue present | Correct |
| GLiDE v1.2 | blue present | Correct |
| Decider v1.1 | blue present | Correct |
| Jev | blue present | Correct |
| Microsoft Decision-1 | red present | Incorrect |
| Clef | red present | Incorrect |
| Luna Decisions | blue present | Correct |
| Clef Flash | red present | Incorrect |
| Strands Decider | blue present | Correct |
| GLiNER2.5 Decide | blue present | Correct |
| Laya | orange ball | Incorrect |
| CalDec v1 | orange ball | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "132",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "ccf444178b2405e2694e1ed226f618c0d6b59eef00ec40a27730c36bc7a81cc7"
}