State tracking and updating · question 46
Back to State tracking and updating · All capabilities
Question ID: bbh/a282bddd396533e6268b
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a red present, Bob has a purple present, Claire has a white present, Dave has a brown present, Eve has a green present, Fred has a blue present, and Gertrude has a yellow present. As the event progresses, pairs of people swap gifts. First, Dave and Bob swap their gifts. Then, Alice and Bob swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Bob and Dave swap their gifts. Then, Fred and Alice swap their gifts. Finally, Fred and Eve swap their gifts. At the end of the event, Eve has the
Answer choices
- A: red present
- B: purple present
- C: white present
- D: brown present
- E: green present
- F: blue present
- G: yellow present
Reference answer: white present (C)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a red present, Bob has a purple present, Claire has a white present, Dave has a brown present, Eve has a green present, Fred has a blue present, and Gertrude has a yellow present.\nAs the event progresses, pairs of people swap gifts. First, Dave and Bob swap their gifts. Then, Alice and Bob swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Bob and Dave swap their gifts. Then, Fred and Alice swap their gifts. Finally, Fred and Eve swap their gifts. At the end of the event, Eve has the",
"choices": {
"A": "red present",
"B": "purple present",
"C": "white present",
"D": "brown present",
"E": "green present",
"F": "blue present",
"G": "yellow present"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | white present | Correct |
| Opus 5.5 | white present | Correct |
| GLiDE v1.2 | white present | Correct |
| Decider v1.1 | yellow present | Incorrect |
| Jev | white present | Correct |
| Microsoft Decision-1 | blue present | Incorrect |
| Clef | blue present | Incorrect |
| Luna Decisions | blue present | Incorrect |
| Clef Flash | blue present | Incorrect |
| Strands Decider | white present | Correct |
| GLiNER2.5 Decide | red present | Incorrect |
| Laya | green present | Incorrect |
| CalDec v1 | green present | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_seven_objects.json",
"record": "116",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "2616600dc5764dd06108f2e69dd9f95a0e13cae6f9a18afc182d104d0f692893"
}