State tracking and updating · question 19
Back to State tracking and updating · All capabilities
Question ID: bbh/639b88b1fe80634a7d66
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a black ball, Bob has a purple present, and Claire has a blue present. As the event progresses, pairs of people swap gifts. First, Claire and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Bob swap their gifts. At the end of the event, Alice has the
Answer choices
- A: black ball
- B: purple present
- C: blue present
Reference answer: black ball (A)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a black ball, Bob has a purple present, and Claire has a blue present.\nAs the event progresses, pairs of people swap gifts. First, Claire and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Bob swap their gifts. At the end of the event, Alice has the",
"choices": {
"A": "black ball",
"B": "purple present",
"C": "blue present"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | black ball | Correct |
| Opus 5.5 | black ball | Correct |
| GLiDE v1.2 | black ball | Correct |
| Decider v1.1 | purple present | Incorrect |
| Jev | blue present | Incorrect |
| Microsoft Decision-1 | blue present | Incorrect |
| Clef | purple present | Incorrect |
| Luna Decisions | purple present | Incorrect |
| Clef Flash | purple present | Incorrect |
| Strands Decider | purple present | Incorrect |
| GLiNER2.5 Decide | black ball | Correct |
| Laya | black ball | Correct |
| CalDec v1 | black ball | Correct |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "115",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "319320542a5390f3b3a04f5a29706d50471235154ef21c7d30277b33f27c2e8a"
}