RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 6

Back to Concepts and categorization · All capabilities

Question ID: mmlu/47c9ae2f980cd5567f31
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

Which of these snakes is poisonous?

Answer choices

  • A: anaconda
  • B: boa constrictor
  • C: copperhead
  • D: python

Reference answer: copperhead (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Which of these snakes is poisonous?",
  "choices": {
    "A": "anaconda",
    "B": "boa constrictor",
    "C": "copperhead",
    "D": "python"
  }
}

Recorded model answers

Model Answer Outcome
Astra copperhead Correct
Opus 5.5 copperhead Correct
GLiDE v1.2 copperhead Correct
Decider v1.1 copperhead Correct
Jev copperhead Correct
Microsoft Decision-1 copperhead Correct
Clef copperhead Correct
Luna Decisions copperhead Correct
Clef Flash copperhead Correct
Strands Decider copperhead Correct
GLiNER2.5 Decide boa constrictor Incorrect
Laya anaconda Incorrect
CalDec v1 python Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "8007",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "3164dafc70f7c954abdfc3d73944bb33bc9ef34b69c5373fbaacae37944c5581"
}

Type to search questions, variants and documentation.