State tracking and updating · question 56
Back to State tracking and updating · All capabilities
Question ID: bbh/236e9e08fe134837a4bc
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a pink ball, Bob has a white present, and Claire has a blue present. As the event progresses, pairs of people swap gifts. First, Bob and Claire swap their gifts. Then, Claire and Alice swap their gifts. Finally, Claire and Bob swap their gifts. At the end of the event, Alice has the
Answer choices
- A: pink ball
- B: white present
- C: blue present
Reference answer: white present (B)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a pink ball, Bob has a white present, and Claire has a blue present.\nAs the event progresses, pairs of people swap gifts. First, Bob and Claire swap their gifts. Then, Claire and Alice swap their gifts. Finally, Claire and Bob swap their gifts. At the end of the event, Alice has the",
"choices": {
"A": "pink ball",
"B": "white present",
"C": "blue present"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | white present | Correct |
| Opus 5.5 | white present | Correct |
| GLiDE v1.2 | white present | Correct |
| Decider v1.1 | white present | Correct |
| Jev | white present | Correct |
| Microsoft Decision-1 | white present | Correct |
| Clef | white present | Correct |
| Luna Decisions | blue present | Incorrect |
| Clef Flash | blue present | Incorrect |
| Strands Decider | white present | Correct |
| GLiNER2.5 Decide | white present | Correct |
| Laya | pink ball | Incorrect |
| CalDec v1 | pink ball | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "27",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "c7d35031cd49146525a9e8305e28d4ff5e15ebb67e778ec5a8b1b178ace7d3a8"
}