RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 59

Back to Relations · All capabilities

Question ID: bbh/449810dd0f48beb00aaf
Source: BIG-Bench Hard selected tasks
Input format: native choices

Explore this question’s altered versions

Question

The following paragraphs each describe a set of five objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells five fruits: watermelons, oranges, loquats, plums, and kiwis. The plums are less expensive than the kiwis. The plums are more expensive than the watermelons. The loquats are more expensive than the kiwis. The oranges are the most expensive.

Answer choices

  • A: The watermelons are the second-most expensive
  • B: The oranges are the second-most expensive
  • C: The loquats are the second-most expensive
  • D: The plums are the second-most expensive
  • E: The kiwis are the second-most expensive

Reference answer: The loquats are the second-most expensive (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The following paragraphs each describe a set of five objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells five fruits: watermelons, oranges, loquats, plums, and kiwis. The plums are less expensive than the kiwis. The plums are more expensive than the watermelons. The loquats are more expensive than the kiwis. The oranges are the most expensive.",
  "choices": {
    "A": "The watermelons are the second-most expensive",
    "B": "The oranges are the second-most expensive",
    "C": "The loquats are the second-most expensive",
    "D": "The plums are the second-most expensive",
    "E": "The kiwis are the second-most expensive"
  }
}

Recorded model answers

Model Answer Outcome
Astra The loquats are the second-most expensive Correct
Opus 5.5 The loquats are the second-most expensive Correct
GLiDE v1.2 The loquats are the second-most expensive Correct
Decider v1.1 The loquats are the second-most expensive Correct
Jev The loquats are the second-most expensive Correct
Microsoft Decision-1 The loquats are the second-most expensive Correct
Clef The loquats are the second-most expensive Correct
Luna Decisions The loquats are the second-most expensive Correct
Clef Flash The loquats are the second-most expensive Correct
Strands Decider The plums are the second-most expensive Incorrect
GLiNER2.5 Decide The plums are the second-most expensive Incorrect
Laya The oranges are the second-most expensive Incorrect
CalDec v1 The oranges are the second-most expensive Incorrect

Source and sharing

Attribution: Mirac Suzgun and BIG-Bench Hard authors; BIG-bench contributors.
Terms: MIT AND Apache-2.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "revision-15-abilities/logical_deduction_five_objects.json",
    "record": "187",
    "kind": "source_file_record"
  },
  "source_revision": "9ee07bd481feebf959a6b59d61ea57bdcf30964d",
  "input_hash": "d1197265d9782a73fbfdfcb0eb0c509dca7df0ef4cf45d01a5c517b62cf9a50c"
}

Type to search questions, variants and documentation.