RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 37

Back to Concepts and categorization · All capabilities

Question ID: mmlu/5562d36bb02e0bb2040a
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I watched the nurse grab a box of crackers and put them in her purse at work. Scenario 2 | I was due to start work soon, so I went there.

Answer choices

  • A: Wrong, Wrong
  • B: Wrong, Not wrong
  • C: Not wrong, Wrong
  • D: Not wrong, Not wrong

Reference answer: Not wrong, Not wrong (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I watched the nurse grab a box of crackers and put them in her purse at work. Scenario 2 | I was due to start work soon, so I went there.",
  "choices": {
    "A": "Wrong, Wrong",
    "B": "Wrong, Not wrong",
    "C": "Not wrong, Wrong",
    "D": "Not wrong, Not wrong"
  }
}

Recorded model answers

Model Answer Outcome
Astra Not wrong, Not wrong Correct
Opus 5.5 Not wrong, Not wrong Correct
GLiDE v1.2 Not wrong, Not wrong Correct
Decider v1.1 Not wrong, Not wrong Correct
Jev Not wrong, Not wrong Correct
Microsoft Decision-1 Not wrong, Not wrong Correct
Clef Not wrong, Not wrong Correct
Luna Decisions Not wrong, Not wrong Correct
Clef Flash Not wrong, Not wrong Correct
Strands Decider Wrong, Not wrong Incorrect
GLiNER2.5 Decide Wrong, Wrong Incorrect
Laya Wrong, Wrong Incorrect
CalDec v1 Wrong, Wrong Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "8519",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "a94db84180f40fe2dda2079be9b74134d65a8cade7d0d0fe334f811e8383a34f"
}

Type to search questions, variants and documentation.