RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 13

Back to Relations · All capabilities

Question ID: bbh/6e437c1f0d8c6a24e020
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells seven fruits: kiwis, cantaloupes, oranges, loquats, pears, watermelons, and peaches. The loquats are less expensive than the peaches. The pears are the third-cheapest. The oranges are less expensive than the cantaloupes. The loquats are more expensive than the watermelons. The peaches are less expensive than the oranges. The kiwis are the second-most expensive.

Answer choices

  • A: The kiwis are the cheapest
  • B: The cantaloupes are the cheapest
  • C: The oranges are the cheapest
  • D: The loquats are the cheapest
  • E: The pears are the cheapest
  • F: The watermelons are the cheapest
  • G: The peaches are the cheapest

Reference answer: The watermelons are the cheapest (F)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells seven fruits: kiwis, cantaloupes, oranges, loquats, pears, watermelons, and peaches. The loquats are less expensive than the peaches. The pears are the third-cheapest. The oranges are less expensive than the cantaloupes. The loquats are more expensive than the watermelons. The peaches are less expensive than the oranges. The kiwis are the second-most expensive.",
  "choices": {
    "A": "The kiwis are the cheapest",
    "B": "The cantaloupes are the cheapest",
    "C": "The oranges are the cheapest",
    "D": "The loquats are the cheapest",
    "E": "The pears are the cheapest",
    "F": "The watermelons are the cheapest",
    "G": "The peaches are the cheapest"
  }
}

Recorded model answers

Model Answer Outcome
Astra The watermelons are the cheapest Correct
Opus 5.5 The watermelons are the cheapest Correct
GLiDE v1.2 The watermelons are the cheapest Correct
Decider v1.1 The watermelons are the cheapest Correct
Jev The watermelons are the cheapest Correct
Microsoft Decision-1 The watermelons are the cheapest Correct
Clef The watermelons are the cheapest Correct
Luna Decisions The watermelons are the cheapest Correct
Clef Flash The watermelons are the cheapest Correct
Strands Decider The watermelons are the cheapest Correct
GLiNER2.5 Decide The peaches are the cheapest Incorrect
Laya The kiwis are the cheapest Incorrect
CalDec v1 The kiwis are the cheapest Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_seven_objects.json",
    "record": "174",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "6c943432e574689dc2c054604788bb872f6f1ff4a801c8ffd7d2f0eb940c791a"
}

Type to search questions, variants and documentation.