State tracking and updating · question 44
Back to State tracking and updating · All capabilities
Question ID: bbh/410fb1d928f65fa33c03
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Catch-22, Bob gets Moby Dick, and Claire gets The Fellowship of the Ring. As the semester proceeds, they start trading around the new books. First, Claire and Alice swap books. Then, Bob and Alice swap books. Finally, Claire and Bob swap books. At the end of the semester, Alice has
Answer choices
- A: Catch-22
- B: Moby Dick
- C: The Fellowship of the Ring
Reference answer: Moby Dick (B)
Exact evaluated input as JSON
{
"state": "",
"question": "Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Catch-22, Bob gets Moby Dick, and Claire gets The Fellowship of the Ring.\nAs the semester proceeds, they start trading around the new books. First, Claire and Alice swap books. Then, Bob and Alice swap books. Finally, Claire and Bob swap books. At the end of the semester, Alice has",
"choices": {
"A": "Catch-22",
"B": "Moby Dick",
"C": "The Fellowship of the Ring"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | Moby Dick | Correct |
| Opus 5.5 | Moby Dick | Correct |
| GLiDE v1.2 | Moby Dick | Correct |
| Decider v1.1 | Moby Dick | Correct |
| Jev | Moby Dick | Correct |
| Microsoft Decision-1 | The Fellowship of the Ring | Incorrect |
| Clef | Moby Dick | Correct |
| Luna Decisions | The Fellowship of the Ring | Incorrect |
| Clef Flash | Moby Dick | Correct |
| Strands Decider | The Fellowship of the Ring | Incorrect |
| GLiNER2.5 Decide | Catch-22 | Incorrect |
| Laya | Catch-22 | Incorrect |
| CalDec v1 | Catch-22 | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "bbh/tracking_shuffled_objects_three_objects.json",
"record": "86",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "461a4d22153f60eabe99fc95b2783506db236a83e9bf0e6143eabf5e9cca784b"
}