RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 51

Back to Relations · All capabilities

Question ID: bbh/ef7b6dcba9cd43e5a5ef
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a convertible, a station wagon, a minivan, a bus, a tractor, and a truck. The minivan is the oldest. The truck is older than the station wagon. The truck is newer than the limousine. The bus is newer than the convertible. The bus is older than the tractor. The limousine is newer than the tractor.

Answer choices

  • A: The limousine is the fourth-newest
  • B: The convertible is the fourth-newest
  • C: The station wagon is the fourth-newest
  • D: The minivan is the fourth-newest
  • E: The bus is the fourth-newest
  • F: The tractor is the fourth-newest
  • G: The truck is the fourth-newest

Reference answer: The tractor is the fourth-newest (F)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a convertible, a station wagon, a minivan, a bus, a tractor, and a truck. The minivan is the oldest. The truck is older than the station wagon. The truck is newer than the limousine. The bus is newer than the convertible. The bus is older than the tractor. The limousine is newer than the tractor.",
  "choices": {
    "A": "The limousine is the fourth-newest",
    "B": "The convertible is the fourth-newest",
    "C": "The station wagon is the fourth-newest",
    "D": "The minivan is the fourth-newest",
    "E": "The bus is the fourth-newest",
    "F": "The tractor is the fourth-newest",
    "G": "The truck is the fourth-newest"
  }
}

Recorded model answers

Model Answer Outcome
Astra The tractor is the fourth-newest Correct
Opus 5.5 The tractor is the fourth-newest Correct
GLiDE v1.2 The limousine is the fourth-newest Incorrect
Decider v1.1 The tractor is the fourth-newest Correct
Jev The tractor is the fourth-newest Correct
Microsoft Decision-1 The limousine is the fourth-newest Incorrect
Clef The tractor is the fourth-newest Correct
Luna Decisions The limousine is the fourth-newest Incorrect
Clef Flash The station wagon is the fourth-newest Incorrect
Strands Decider The station wagon is the fourth-newest Incorrect
GLiNER2.5 Decide The minivan is the fourth-newest Incorrect
Laya The truck is the fourth-newest Incorrect
CalDec v1 The bus is the fourth-newest Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_seven_objects.json",
    "record": "176",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "4aa5af7f9e7861fc5e483d4c3e467e8bb1a8782c9ed7e33efb7d0bffab53c24a"
}

Type to search questions, variants and documentation.