RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 47

Back to Relations · All capabilities

Question ID: bbh/46751a897dac764c632b
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are three vehicles: a tractor, a truck, and a minivan. The minivan is newer than the truck. The tractor is older than the truck.

Answer choices

  • A: The tractor is the newest
  • B: The truck is the newest
  • C: The minivan is the newest

Reference answer: The minivan is the newest (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are three vehicles: a tractor, a truck, and a minivan. The minivan is newer than the truck. The tractor is older than the truck.",
  "choices": {
    "A": "The tractor is the newest",
    "B": "The truck is the newest",
    "C": "The minivan is the newest"
  }
}

Recorded model answers

Model Answer Outcome
Astra The minivan is the newest Correct
Opus 5.5 The minivan is the newest Correct
GLiDE v1.2 The minivan is the newest Correct
Decider v1.1 The minivan is the newest Correct
Jev The minivan is the newest Correct
Microsoft Decision-1 The minivan is the newest Correct
Clef The minivan is the newest Correct
Luna Decisions The minivan is the newest Correct
Clef Flash The minivan is the newest Correct
Strands Decider The minivan is the newest Correct
GLiNER2.5 Decide The minivan is the newest Correct
Laya The truck is the newest Incorrect
CalDec v1 The minivan is the newest Correct

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_three_objects.json",
    "record": "6",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "7a4dfb1dbd19443a4735d46c3077ba08947614fc0a07591d1a38853aae03daee"
}

Type to search questions, variants and documentation.