RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 44

Back to State tracking and updating · All capabilities

Question ID: bbh/410fb1d928f65fa33c03
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Catch-22, Bob gets Moby Dick, and Claire gets The Fellowship of the Ring. As the semester proceeds, they start trading around the new books. First, Claire and Alice swap books. Then, Bob and Alice swap books. Finally, Claire and Bob swap books. At the end of the semester, Alice has

Answer choices

  • A: Catch-22
  • B: Moby Dick
  • C: The Fellowship of the Ring

Reference answer: Moby Dick (B)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Catch-22, Bob gets Moby Dick, and Claire gets The Fellowship of the Ring.\nAs the semester proceeds, they start trading around the new books. First, Claire and Alice swap books. Then, Bob and Alice swap books. Finally, Claire and Bob swap books. At the end of the semester, Alice has",
  "choices": {
    "A": "Catch-22",
    "B": "Moby Dick",
    "C": "The Fellowship of the Ring"
  }
}

Recorded model answers

Model Answer Outcome
Astra Moby Dick Correct
Opus 5.5 Moby Dick Correct
GLiDE v1.2 Moby Dick Correct
Decider v1.1 Moby Dick Correct
Jev Moby Dick Correct
Microsoft Decision-1 The Fellowship of the Ring Incorrect
Clef Moby Dick Correct
Luna Decisions The Fellowship of the Ring Incorrect
Clef Flash Moby Dick Correct
Strands Decider The Fellowship of the Ring Incorrect
GLiNER2.5 Decide Catch-22 Incorrect
Laya Catch-22 Incorrect
CalDec v1 Catch-22 Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_three_objects.json",
    "record": "86",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "461a4d22153f60eabe99fc95b2783506db236a83e9bf0e6143eabf5e9cca784b"
}

Type to search questions, variants and documentation.