RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 45

Back to Relations · All capabilities

Question ID: bbh/61dde6d28af08d7801de
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were seven golfers: Ana, Rob, Eve, Mel, Mya, Joe, and Eli. Joe finished second-to-last. Eli finished first. Mya finished below Joe. Mel finished second. Rob finished third-to-last. Ana finished third.

Answer choices

  • A: Ana finished fourth
  • B: Rob finished fourth
  • C: Eve finished fourth
  • D: Mel finished fourth
  • E: Mya finished fourth
  • F: Joe finished fourth
  • G: Eli finished fourth

Reference answer: Eve finished fourth (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were seven golfers: Ana, Rob, Eve, Mel, Mya, Joe, and Eli. Joe finished second-to-last. Eli finished first. Mya finished below Joe. Mel finished second. Rob finished third-to-last. Ana finished third.",
  "choices": {
    "A": "Ana finished fourth",
    "B": "Rob finished fourth",
    "C": "Eve finished fourth",
    "D": "Mel finished fourth",
    "E": "Mya finished fourth",
    "F": "Joe finished fourth",
    "G": "Eli finished fourth"
  }
}

Recorded model answers

Model Answer Outcome
Astra Eve finished fourth Correct
Opus 5.5 Eve finished fourth Correct
GLiDE v1.2 Eve finished fourth Correct
Decider v1.1 Eve finished fourth Correct
Jev Eve finished fourth Correct
Microsoft Decision-1 Rob finished fourth Incorrect
Clef Eve finished fourth Correct
Luna Decisions Eve finished fourth Correct
Clef Flash Eve finished fourth Correct
Strands Decider Eve finished fourth Correct
GLiNER2.5 Decide Joe finished fourth Incorrect
Laya Joe finished fourth Incorrect
CalDec v1 Eli finished fourth Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_seven_objects.json",
    "record": "182",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "10157423ee8d2d897b5a36db6bc1d0a140fb1d4593e7df671617dc0087b2a618"
}

Type to search questions, variants and documentation.