RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 12

Back to State tracking and updating · All capabilities

Question ID: bbh/11517ef69a2d8f856467
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, Claire, Dave, and Eve are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a purple present, Bob has a blue present, Claire has a brown present, Dave has a yellow present, and Eve has a black ball. As the event progresses, pairs of people swap gifts. First, Alice and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Claire and Bob swap their gifts. Then, Dave and Alice swap their gifts. Finally, Dave and Eve swap their gifts. At the end of the event, Dave has the

Answer choices

  • A: purple present
  • B: blue present
  • C: brown present
  • D: yellow present
  • E: black ball

Reference answer: black ball (E)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, Claire, Dave, and Eve are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a purple present, Bob has a blue present, Claire has a brown present, Dave has a yellow present, and Eve has a black ball.\nAs the event progresses, pairs of people swap gifts. First, Alice and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Claire and Bob swap their gifts. Then, Dave and Alice swap their gifts. Finally, Dave and Eve swap their gifts. At the end of the event, Dave has the",
  "choices": {
    "A": "purple present",
    "B": "blue present",
    "C": "brown present",
    "D": "yellow present",
    "E": "black ball"
  }
}

Recorded model answers

Model Answer Outcome
Astra black ball Correct
Opus 5.5 black ball Correct
GLiDE v1.2 black ball Correct
Decider v1.1 black ball Correct
Jev black ball Correct
Microsoft Decision-1 black ball Correct
Clef black ball Correct
Luna Decisions purple present Incorrect
Clef Flash black ball Correct
Strands Decider purple present Incorrect
GLiNER2.5 Decide purple present Incorrect
Laya yellow present Incorrect
CalDec v1 yellow present Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_five_objects.json",
    "record": "54",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "35684e1e6a2561f8c9d7d155f9a104e75926be09906be6e6573bcc2958c7ba53"
}

Type to search questions, variants and documentation.