Relations · question 51
Back to Relations · All capabilities
Question ID: bbh/ef7b6dcba9cd43e5a5ef
Source: BIG-Bench Hard selected tasks
Input format: native choices
Explore this question’s altered versions
Question
The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a convertible, a station wagon, a minivan, a bus, a tractor, and a truck. The minivan is the oldest. The truck is older than the station wagon. The truck is newer than the limousine. The bus is newer than the convertible. The bus is older than the tractor. The limousine is newer than the tractor.
Answer choices
- A: The limousine is the fourth-newest
- B: The convertible is the fourth-newest
- C: The station wagon is the fourth-newest
- D: The minivan is the fourth-newest
- E: The bus is the fourth-newest
- F: The tractor is the fourth-newest
- G: The truck is the fourth-newest
Reference answer: The tractor is the fourth-newest (F)
Exact evaluated input as JSON
{
"state": "",
"question": "The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. In an antique car show, there are seven vehicles: a limousine, a convertible, a station wagon, a minivan, a bus, a tractor, and a truck. The minivan is the oldest. The truck is older than the station wagon. The truck is newer than the limousine. The bus is newer than the convertible. The bus is older than the tractor. The limousine is newer than the tractor.",
"choices": {
"A": "The limousine is the fourth-newest",
"B": "The convertible is the fourth-newest",
"C": "The station wagon is the fourth-newest",
"D": "The minivan is the fourth-newest",
"E": "The bus is the fourth-newest",
"F": "The tractor is the fourth-newest",
"G": "The truck is the fourth-newest"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | The tractor is the fourth-newest | Correct |
| Opus 5.5 | The tractor is the fourth-newest | Correct |
| GLiDE v1.2 | The limousine is the fourth-newest | Incorrect |
| Decider v1.1 | The tractor is the fourth-newest | Correct |
| Jev | The tractor is the fourth-newest | Correct |
| Microsoft Decision-1 | The limousine is the fourth-newest | Incorrect |
| Clef | The tractor is the fourth-newest | Correct |
| Luna Decisions | The limousine is the fourth-newest | Incorrect |
| Clef Flash | The station wagon is the fourth-newest | Incorrect |
| Strands Decider | The station wagon is the fourth-newest | Incorrect |
| GLiNER2.5 Decide | The minivan is the fourth-newest | Incorrect |
| Laya | The truck is the fourth-newest | Incorrect |
| CalDec v1 | The bus is the fourth-newest | Incorrect |
Source and sharing
Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "revision-15-abilities/logical_deduction_seven_objects.json",
"record": "176",
"kind": "source_file_record"
},
"source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
"input_hash": "4aa5af7f9e7861fc5e483d4c3e467e8bb1a8782c9ed7e33efb7d0bffab53c24a"
}