RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 21

Back to Relations · All capabilities

Question ID: bbh/6b4488eb840e194769e7
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were three golfers: Ana, Mel, and Ada. Ada finished above Ana. Mel finished below Ana.

Answer choices

  • A: Ana finished first
  • B: Mel finished first
  • C: Ada finished first

Reference answer: Ada finished first (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. In a golf tournament, there were three golfers: Ana, Mel, and Ada. Ada finished above Ana. Mel finished below Ana.",
  "choices": {
    "A": "Ana finished first",
    "B": "Mel finished first",
    "C": "Ada finished first"
  }
}

Recorded model answers

Model Answer Outcome
Astra Ada finished first Correct
Opus 5.5 Ada finished first Correct
GLiDE v1.2 Ada finished first Correct
Decider v1.1 Ada finished first Correct
Jev Ada finished first Correct
Microsoft Decision-1 Ada finished first Correct
Clef Ada finished first Correct
Luna Decisions Ada finished first Correct
Clef Flash Ada finished first Correct
Strands Decider Ada finished first Correct
GLiNER2.5 Decide Ada finished first Correct
Laya Ana finished first Incorrect
CalDec v1 Ana finished first Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_three_objects.json",
    "record": "47",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "15b1af58b391bf7fc216664846f2ab15788154962209ff18afbb9633d19b1abd"
}

Type to search questions, variants and documentation.