State tracking and updating · question 30
Back to State tracking and updating · All capabilities
Question ID: bbh/7e2477bff4b9f9a125b9
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Frankenstein, Bob gets Ulysses, Claire gets Lolita, Dave gets The Odyssey, and Eve gets The Fellowship of the Ring. As the semester proceeds, they start trading around the new books. First, Eve and Dave swap books. Then, Claire and Alice swap books. Then, Dave and Eve swap books. Then, Bob and Claire swap books. Finally, Alice and Claire swap books. At the end of the semester, Alice has
Answer choices
- A: Frankenstein
- B: Ulysses
- C: Lolita
- D: The Odyssey
- E: The Fellowship of the Ring
Reference answer: Ulysses (B)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Frankenstein, Bob gets Ulysses, Claire gets Lolita, Dave gets The Odyssey, and Eve gets The Fellowship of the Ring.\nAs the semester proceeds, they start trading around the new books. First, Eve and Dave swap books. Then, Claire and Alice swap books. Then, Dave and Eve swap books. Then, Bob and Claire swap books. Finally, Alice and Claire swap books. At the end of the semester, Alice has",
"choices": {
"A": "Frankenstein",
"B": "Ulysses",
"C": "Lolita",
"D": "The Odyssey",
"E": "The Fellowship of the Ring"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Ulysses | Correct |
| Opus 5.5 | Ulysses | Correct |
| GLiDE v1.2 | Ulysses | Correct |
| Decider v1.1 | Ulysses | Correct |
| Jev | Ulysses | Correct |
| Microsoft Decision-1 | Lolita | Incorrect |
| Clef | Lolita | Incorrect |
| Luna Decisions | Frankenstein | Incorrect |
| Clef Flash | Lolita | Incorrect |
| Strands Decider | Lolita | Incorrect |
| GLiNER2.5 Decide | Frankenstein | Incorrect |
| Laya | Frankenstein | Incorrect |
| CalDec v1 | Frankenstein | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_five_objects.json",
"record": "136",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "ddf72f09c8a44de13e591ee74177aabb320e3391867810df4742d2168345c8ab"
}