RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 30

Back to State tracking and updating · All capabilities

Question ID: bbh/7e2477bff4b9f9a125b9
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Frankenstein, Bob gets Ulysses, Claire gets Lolita, Dave gets The Odyssey, and Eve gets The Fellowship of the Ring. As the semester proceeds, they start trading around the new books. First, Eve and Dave swap books. Then, Claire and Alice swap books. Then, Dave and Eve swap books. Then, Bob and Claire swap books. Finally, Alice and Claire swap books. At the end of the semester, Alice has

Answer choices

  • A: Frankenstein
  • B: Ulysses
  • C: Lolita
  • D: The Odyssey
  • E: The Fellowship of the Ring

Reference answer: Ulysses (B)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Frankenstein, Bob gets Ulysses, Claire gets Lolita, Dave gets The Odyssey, and Eve gets The Fellowship of the Ring.\nAs the semester proceeds, they start trading around the new books. First, Eve and Dave swap books. Then, Claire and Alice swap books. Then, Dave and Eve swap books. Then, Bob and Claire swap books. Finally, Alice and Claire swap books. At the end of the semester, Alice has",
  "choices": {
    "A": "Frankenstein",
    "B": "Ulysses",
    "C": "Lolita",
    "D": "The Odyssey",
    "E": "The Fellowship of the Ring"
  }
}

Recorded model answers

Model Answer Outcome
Astra Ulysses Correct
Opus 5.5 Ulysses Correct
GLiDE v1.2 Ulysses Correct
Decider v1.1 Ulysses Correct
Jev Ulysses Correct
Microsoft Decision-1 Lolita Incorrect
Clef Lolita Incorrect
Luna Decisions Frankenstein Incorrect
Clef Flash Lolita Incorrect
Strands Decider Lolita Incorrect
GLiNER2.5 Decide Frankenstein Incorrect
Laya Frankenstein Incorrect
CalDec v1 Frankenstein Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_five_objects.json",
    "record": "136",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "ddf72f09c8a44de13e591ee74177aabb320e3391867810df4742d2168345c8ab"
}

Type to search questions, variants and documentation.