RaindropA practical guide to choosing decision modelsGitHub ↗

Rule application · question 9

Back to Rule application · All capabilities

Question ID: mmlu/372a72b278f6e7e4df70
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

S(n) is a statement about positive integers n such that whenever S(k) is true, S(k + 1) must also be true. Furthermore, there exists some positive integer n0 such that S(n0) is not true. Of the following, which is the strongest conclusion that can be drawn?

Answer choices

  • A: S(n0 + 1) is not true.
  • B: S(n0 - 1) is not true.
  • C: S(n) is not true for any n <= n0
  • D: S(n) is not true for any n >= n0

Reference answer: S(n) is not true for any n <= n0 (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "S(n) is a statement about positive integers n such that whenever S(k) is true, S(k + 1) must also be true. Furthermore, there exists some positive integer n0 such that S(n0) is not true. Of the following, which is the strongest conclusion that can be drawn?",
  "choices": {
    "A": "S(n0 + 1) is not true.",
    "B": "S(n0 - 1) is not true.",
    "C": "S(n) is not true for any n <= n0",
    "D": "S(n) is not true for any n >= n0"
  }
}

Recorded model answers

Model Answer Outcome
Astra S(n) is not true for any n <= n0 Correct
Opus 5.5 S(n) is not true for any n <= n0 Correct
GLiDE v1.2 S(n) is not true for any n >= n0 Incorrect
Decider v1.1 S(n) is not true for any n <= n0 Correct
Jev S(n) is not true for any n <= n0 Correct
Microsoft Decision-1 S(n) is not true for any n >= n0 Incorrect
Clef S(n) is not true for any n <= n0 Correct
Luna Decisions S(n0 + 1) is not true. Incorrect
Clef Flash S(n) is not true for any n <= n0 Correct
Strands Decider S(n) is not true for any n >= n0 Incorrect
GLiNER2.5 Decide S(n) is not true for any n >= n0 Incorrect
Laya S(n) is not true for any n <= n0 Correct
CalDec v1 S(n0 + 1) is not true. Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "1126",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "70adfeb43fc0706147c987d14e132bd31b9e0a50832a86f166059a804fa3e27b"
}

Type to search questions, variants and documentation.