RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 46

Back to State tracking and updating · All capabilities

Question ID: bbh/a282bddd396533e6268b
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a red present, Bob has a purple present, Claire has a white present, Dave has a brown present, Eve has a green present, Fred has a blue present, and Gertrude has a yellow present. As the event progresses, pairs of people swap gifts. First, Dave and Bob swap their gifts. Then, Alice and Bob swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Bob and Dave swap their gifts. Then, Fred and Alice swap their gifts. Finally, Fred and Eve swap their gifts. At the end of the event, Eve has the

Answer choices

  • A: red present
  • B: purple present
  • C: white present
  • D: brown present
  • E: green present
  • F: blue present
  • G: yellow present

Reference answer: white present (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a red present, Bob has a purple present, Claire has a white present, Dave has a brown present, Eve has a green present, Fred has a blue present, and Gertrude has a yellow present.\nAs the event progresses, pairs of people swap gifts. First, Dave and Bob swap their gifts. Then, Alice and Bob swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Bob and Dave swap their gifts. Then, Fred and Alice swap their gifts. Finally, Fred and Eve swap their gifts. At the end of the event, Eve has the",
  "choices": {
    "A": "red present",
    "B": "purple present",
    "C": "white present",
    "D": "brown present",
    "E": "green present",
    "F": "blue present",
    "G": "yellow present"
  }
}

Recorded model answers

Model Answer Outcome
Astra white present Correct
Opus 5.5 white present Correct
GLiDE v1.2 white present Correct
Decider v1.1 yellow present Incorrect
Jev white present Correct
Microsoft Decision-1 blue present Incorrect
Clef blue present Incorrect
Luna Decisions blue present Incorrect
Clef Flash blue present Incorrect
Strands Decider white present Correct
GLiNER2.5 Decide red present Incorrect
Laya green present Incorrect
CalDec v1 green present Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_seven_objects.json",
    "record": "116",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "2616600dc5764dd06108f2e69dd9f95a0e13cae6f9a18afc182d104d0f692893"
}

Type to search questions, variants and documentation.