RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 19

Back to State tracking and updating · All capabilities

Question ID: bbh/639b88b1fe80634a7d66
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a black ball, Bob has a purple present, and Claire has a blue present. As the event progresses, pairs of people swap gifts. First, Claire and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Bob swap their gifts. At the end of the event, Alice has the

Answer choices

  • A: black ball
  • B: purple present
  • C: blue present

Reference answer: black ball (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a black ball, Bob has a purple present, and Claire has a blue present.\nAs the event progresses, pairs of people swap gifts. First, Claire and Alice swap their gifts. Then, Bob and Claire swap their gifts. Finally, Alice and Bob swap their gifts. At the end of the event, Alice has the",
  "choices": {
    "A": "black ball",
    "B": "purple present",
    "C": "blue present"
  }
}

Recorded model answers

Model Answer Outcome
Astra black ball Correct
Opus 5.5 black ball Correct
GLiDE v1.2 black ball Correct
Decider v1.1 purple present Incorrect
Jev blue present Incorrect
Microsoft Decision-1 blue present Incorrect
Clef purple present Incorrect
Luna Decisions purple present Incorrect
Clef Flash purple present Incorrect
Strands Decider purple present Incorrect
GLiNER2.5 Decide black ball Correct
Laya black ball Correct
CalDec v1 black ball Correct

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_three_objects.json",
    "record": "115",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "319320542a5390f3b3a04f5a29706d50471235154ef21c7d30277b33f27c2e8a"
}

Type to search questions, variants and documentation.