RaindropA practical guide to choosing decision modelsGitHub ↗

Rule application · question 6

Back to Rule application · All capabilities

Question ID: mmlu/2e3b68a90b4764e3379b
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

Select the best English interpretation of the given proposition, using the following translation key: Ax: x is an apartment Hx: x is a house Lx: x is large Bxy: x is bigger than y (∀x){(Lx • Ax) ⊃ (∃y)[(Hy • ~Ly) • Bxy]}

Answer choices

  • A: All large apartments are bigger than all houses that are not large.
  • B: Some house that is not large is bigger than all large apartments.
  • C: Any apartment bigger than a house that is not large is large.
  • D: Every large apartment is bigger than some house that is not large.

Reference answer: Every large apartment is bigger than some house that is not large. (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": " Select the best English interpretation of the given proposition, using the following translation key: Ax: x is an apartment Hx: x is a house Lx: x is large Bxy: x is bigger than y (∀x){(Lx • Ax) ⊃ (∃y)[(Hy • ~Ly) • Bxy]}",
  "choices": {
    "A": "All large apartments are bigger than all houses that are not large.",
    "B": "Some house that is not large is bigger than all large apartments.",
    "C": "Any apartment bigger than a house that is not large is large.",
    "D": "Every large apartment is bigger than some house that is not large."
  }
}

Recorded model answers

Model Answer Outcome
Astra Every large apartment is bigger than some house that is not large. Correct
Opus 5.5 Every large apartment is bigger than some house that is not large. Correct
GLiDE v1.2 Every large apartment is bigger than some house that is not large. Correct
Decider v1.1 Every large apartment is bigger than some house that is not large. Correct
Jev Every large apartment is bigger than some house that is not large. Correct
Microsoft Decision-1 Every large apartment is bigger than some house that is not large. Correct
Clef Every large apartment is bigger than some house that is not large. Correct
Luna Decisions Every large apartment is bigger than some house that is not large. Correct
Clef Flash Every large apartment is bigger than some house that is not large. Correct
Strands Decider Every large apartment is bigger than some house that is not large. Correct
GLiNER2.5 Decide All large apartments are bigger than all houses that are not large. Incorrect
Laya Any apartment bigger than a house that is not large is large. Incorrect
CalDec v1 Every large apartment is bigger than some house that is not large. Correct

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/validation-00000-of-00001.parquet",
    "record": "272",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "fc6cb9d7a9304ab8187799b9589100b28d7b438953b9a6c3101372c042a7f6d1"
}

Type to search questions, variants and documentation.