RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 57

Back to Relations · All capabilities

Question ID: bbh/ff740484ebcc04612637
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of five objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were five golfers: Dan, Ana, Eli, Mya, and Amy. Amy finished above Eli. Ana finished above Amy. Mya finished below Eli. Dan finished last.

Answer choices

  • A: Dan finished third
  • B: Ana finished third
  • C: Eli finished third
  • D: Mya finished third
  • E: Amy finished third

Reference answer: Eli finished third (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of five objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were five golfers: Dan, Ana, Eli, Mya, and Amy. Amy finished above Eli. Ana finished above Amy. Mya finished below Eli. Dan finished last.",
  "choices": {
    "A": "Dan finished third",
    "B": "Ana finished third",
    "C": "Eli finished third",
    "D": "Mya finished third",
    "E": "Amy finished third"
  }
}

Recorded model answers

Model Answer Outcome
Astra Eli finished third Correct
Opus 5.5 Eli finished third Correct
GLiDE v1.2 Eli finished third Correct
Decider v1.1 Eli finished third Correct
Jev Eli finished third Correct
Microsoft Decision-1 Amy finished third Incorrect
Clef Mya finished third Incorrect
Luna Decisions Mya finished third Incorrect
Clef Flash Eli finished third Correct
Strands Decider Ana finished third Incorrect
GLiNER2.5 Decide Amy finished third Incorrect
Laya Amy finished third Incorrect
CalDec v1 Ana finished third Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_five_objects.json",
    "record": "92",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "edbe83d6e1da3b6f636bb0c5529fea4454a77c5166a8c4a60ae816b75ba07ae0"
}

Type to search questions, variants and documentation.