RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 24

Back to State tracking and updating · All capabilities

Question ID: bbh/deb22500aa23655847cc
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing right winger, Bob is playing right midfielder, Claire is playing left winger, Dave is playing cheerleader, and Eve is playing goalkeeper. As the game progresses, pairs of players occasionally swap positions. First, Dave and Bob trade positions. Then, Dave and Eve trade positions. Then, Alice and Bob trade positions. Then, Dave and Claire trade positions. Finally, Dave and Eve trade positions. At the end of the match, Eve is playing

Answer choices

  • A: right winger
  • B: right midfielder
  • C: left winger
  • D: cheerleader
  • E: goalkeeper

Reference answer: left winger (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing right winger, Bob is playing right midfielder, Claire is playing left winger, Dave is playing cheerleader, and Eve is playing goalkeeper.\nAs the game progresses, pairs of players occasionally swap positions. First, Dave and Bob trade positions. Then, Dave and Eve trade positions. Then, Alice and Bob trade positions. Then, Dave and Claire trade positions. Finally, Dave and Eve trade positions. At the end of the match, Eve is playing",
  "choices": {
    "A": "right winger",
    "B": "right midfielder",
    "C": "left winger",
    "D": "cheerleader",
    "E": "goalkeeper"
  }
}

Recorded model answers

Model Answer Outcome
Astra left winger Correct
Opus 5.5 left winger Correct
GLiDE v1.2 left winger Correct
Decider v1.1 left winger Correct
Jev left winger Correct
Microsoft Decision-1 left winger Correct
Clef cheerleader Incorrect
Luna Decisions cheerleader Incorrect
Clef Flash cheerleader Incorrect
Strands Decider goalkeeper Incorrect
GLiNER2.5 Decide goalkeeper Incorrect
Laya goalkeeper Incorrect
CalDec v1 goalkeeper Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_five_objects.json",
    "record": "143",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "1ef2cb27ec66e21bd46e40411913249372682d62296bc9d719dd8b750b02f6d4"
}

Type to search questions, variants and documentation.