RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 9

Back to Relations · All capabilities

Question ID: bbh/a3c4787ea5640260cdd0
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a sedan, a tractor, a motorcyle, a minivan, a hatchback, and a truck. The minivan is newer than the limousine. The tractor is the second-newest. The truck is older than the sedan. The minivan is older than the truck. The hatchback is newer than the tractor. The motorcyle is the fourth-newest.

Answer choices

  • A: The limousine is the oldest
  • B: The sedan is the oldest
  • C: The tractor is the oldest
  • D: The motorcyle is the oldest
  • E: The minivan is the oldest
  • F: The hatchback is the oldest
  • G: The truck is the oldest

Reference answer: The limousine is the oldest (A)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a sedan, a tractor, a motorcyle, a minivan, a hatchback, and a truck. The minivan is newer than the limousine. The tractor is the second-newest. The truck is older than the sedan. The minivan is older than the truck. The hatchback is newer than the tractor. The motorcyle is the fourth-newest.",
  "choices": {
    "A": "The limousine is the oldest",
    "B": "The sedan is the oldest",
    "C": "The tractor is the oldest",
    "D": "The motorcyle is the oldest",
    "E": "The minivan is the oldest",
    "F": "The hatchback is the oldest",
    "G": "The truck is the oldest"
  }
}

Recorded model answers

Model Answer Outcome
Astra The limousine is the oldest Correct
Opus 5.5 The limousine is the oldest Correct
GLiDE v1.2 The limousine is the oldest Correct
Decider v1.1 The limousine is the oldest Correct
Jev The limousine is the oldest Correct
Microsoft Decision-1 The limousine is the oldest Correct
Clef The limousine is the oldest Correct
Luna Decisions The sedan is the oldest Incorrect
Clef Flash The limousine is the oldest Correct
Strands Decider The limousine is the oldest Correct
GLiNER2.5 Decide The minivan is the oldest Incorrect
Laya The minivan is the oldest Incorrect
CalDec v1 The minivan is the oldest Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_seven_objects.json",
    "record": "100",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "776d7d36a03a293f2467b53991e435b0e9ed5a6ecbee1c6cc604828b058f622b"
}

Type to search questions, variants and documentation.