RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 14

Back to State tracking and updating · All capabilities

Question ID: bbh/2d455db1ae23f83bf568
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a green ball, and Claire has a blue ball. As the game progresses, pairs of players trade balls. First, Alice and Bob swap balls. Then, Alice and Claire swap balls. Finally, Claire and Bob swap balls. At the end of the game, Alice has the

Answer choices

  • A: yellow ball
  • B: green ball
  • C: blue ball

Reference answer: blue ball (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a green ball, and Claire has a blue ball.\nAs the game progresses, pairs of players trade balls. First, Alice and Bob swap balls. Then, Alice and Claire swap balls. Finally, Claire and Bob swap balls. At the end of the game, Alice has the",
  "choices": {
    "A": "yellow ball",
    "B": "green ball",
    "C": "blue ball"
  }
}

Recorded model answers

Model Answer Outcome
Astra blue ball Correct
Opus 5.5 blue ball Correct
GLiDE v1.2 blue ball Correct
Decider v1.1 blue ball Correct
Jev blue ball Correct
Microsoft Decision-1 green ball Incorrect
Clef blue ball Correct
Luna Decisions green ball Incorrect
Clef Flash green ball Incorrect
Strands Decider green ball Incorrect
GLiNER2.5 Decide yellow ball Incorrect
Laya yellow ball Incorrect
CalDec v1 yellow ball Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_three_objects.json",
    "record": "159",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "1837053820f5f2fc37301ebae39c0df53d023ec498fcfbc66a511e258b960917"
}

Type to search questions, variants and documentation.