RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 8

Back to State tracking and updating · All capabilities

Question ID: bbh/663e1c7714920e03d8cd
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

Alice, Bob, Claire, Dave, and Eve are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Sam, Bob is dancing with Izzi, Claire is dancing with Karl, Dave is dancing with Helga, and Eve is dancing with Ophelia. Throughout the song, the dancers often trade partners. First, Bob and Alice switch partners. Then, Alice and Claire switch partners. Then, Alice and Dave switch partners. Then, Alice and Eve switch partners. Finally, Dave and Claire switch partners. At the end of the dance, Claire is dancing with

Answer choices

  • A: Sam
  • B: Izzi
  • C: Karl
  • D: Helga
  • E: Ophelia

Reference answer: Karl (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Alice, Bob, Claire, Dave, and Eve are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Sam, Bob is dancing with Izzi, Claire is dancing with Karl, Dave is dancing with Helga, and Eve is dancing with Ophelia.\nThroughout the song, the dancers often trade partners. First, Bob and Alice switch partners. Then, Alice and Claire switch partners. Then, Alice and Dave switch partners. Then, Alice and Eve switch partners. Finally, Dave and Claire switch partners. At the end of the dance, Claire is dancing with",
  "choices": {
    "A": "Sam",
    "B": "Izzi",
    "C": "Karl",
    "D": "Helga",
    "E": "Ophelia"
  }
}

Recorded model answers

Model Answer Outcome
Astra Karl Correct
Opus 5.5 Karl Correct
GLiDE v1.2 Helga Incorrect
Decider v1.1 Ophelia Incorrect
Jev Karl Correct
Microsoft Decision-1 Helga Incorrect
Clef Sam Incorrect
Luna Decisions Helga Incorrect
Clef Flash Helga Incorrect
Strands Decider Helga Incorrect
GLiNER2.5 Decide Ophelia Incorrect
Laya Sam Incorrect
CalDec v1 Sam Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "bbh/tracking_shuffled_objects_five_objects.json",
    "record": "74",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "1bb8b1594d98e7ef25fee0578d2c1ee3caf9ea5e7552694f5724b3e0c3327675"
}

Type to search questions, variants and documentation.