State tracking and updating · question 62
Back to State tracking and updating · All capabilities
Question ID: bbh/e9d5fa0fe20ed7cd4ead
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a orange ball, Bob has a green present, and Claire has a red present. As the event progresses, pairs of people swap gifts. First, Bob and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Claire swap their gifts. At the end of the event, Alice has the
Answer choices
- A: orange ball
- B: green present
- C: red present
Reference answer: orange ball (A)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a orange ball, Bob has a green present, and Claire has a red present.\nAs the event progresses, pairs of people swap gifts. First, Bob and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Claire swap their gifts. At the end of the event, Alice has the",
"choices": {
"A": "orange ball",
"B": "green present",
"C": "red present"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | orange ball | Correct |
| Opus 5.5 | orange ball | Correct |
| GLiDE v1.2 | orange ball | Correct |
| Decider v1.1 | red present | Incorrect |
| Jev | orange ball | Correct |
| Microsoft Decision-1 | orange ball | Correct |
| Clef | red present | Incorrect |
| Luna Decisions | green present | Incorrect |
| Clef Flash | red present | Incorrect |
| Strands Decider | red present | Incorrect |
| GLiNER2.5 Decide | orange ball | Correct |
| Laya | orange ball | Correct |
| CalDec v1 | orange ball | Correct |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "230",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "061f697b1d3395c34adfef22f128581fa301c8442253ce03bd67e386d781191e"
}