State tracking and updating · question 14
Back to State tracking and updating · All capabilities
Question ID: bbh/2d455db1ae23f83bf568
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a green ball, and Claire has a blue ball. As the game progresses, pairs of players trade balls. First, Alice and Bob swap balls. Then, Alice and Claire swap balls. Finally, Claire and Bob swap balls. At the end of the game, Alice has the
Answer choices
- A: yellow ball
- B: green ball
- C: blue ball
Reference answer: blue ball (C)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a green ball, and Claire has a blue ball.\nAs the game progresses, pairs of players trade balls. First, Alice and Bob swap balls. Then, Alice and Claire swap balls. Finally, Claire and Bob swap balls. At the end of the game, Alice has the",
"choices": {
"A": "yellow ball",
"B": "green ball",
"C": "blue ball"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | blue ball | Correct |
| Opus 5.5 | blue ball | Correct |
| GLiDE v1.2 | blue ball | Correct |
| Decider v1.1 | blue ball | Correct |
| Jev | blue ball | Correct |
| Microsoft Decision-1 | green ball | Incorrect |
| Clef | blue ball | Correct |
| Luna Decisions | green ball | Incorrect |
| Clef Flash | green ball | Incorrect |
| Strands Decider | green ball | Incorrect |
| GLiNER2.5 Decide | yellow ball | Incorrect |
| Laya | yellow ball | Incorrect |
| CalDec v1 | yellow ball | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "159",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "1837053820f5f2fc37301ebae39c0df53d023ec498fcfbc66a511e258b960917"
}