RaindropA practical guide to choosing decision modelsGitHub ↗

Numerical cognition · question 28

Back to Numerical cognition · All capabilities

Question ID: mmlu/af960c89b8a848cc715f
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

The American flag is usually made with its width and length in the ratio of 10 to 19. Which dimensions are in the correct ratio for the flag?

Answer choices

  • A: 30 ft by 53 ft
  • B: 27 ft by 53 ft
  • C: 27 in. by 57 in.
  • D: 30 in. by 57 in.

Reference answer: 30 in. by 57 in. (D)

Exact evaluated input as JSON
{
  "state": "",
  "question": "The American flag is usually made with its width and length in the ratio of 10 to 19. Which dimensions are in the correct ratio for the flag?",
  "choices": {
    "A": "30 ft by 53 ft",
    "B": "27 ft by 53 ft",
    "C": "27 in. by 57 in.",
    "D": "30 in. by 57 in."
  }
}

Recorded model answers

Model Answer Outcome
Astra 30 in. by 57 in. Correct
Opus 5.5 30 in. by 57 in. Correct
GLiDE v1.2 30 in. by 57 in. Correct
Decider v1.1 30 in. by 57 in. Correct
Jev 30 in. by 57 in. Correct
Microsoft Decision-1 30 in. by 57 in. Correct
Clef 30 in. by 57 in. Correct
Luna Decisions 30 in. by 57 in. Correct
Clef Flash 30 in. by 57 in. Correct
Strands Decider 30 in. by 57 in. Correct
GLiNER2.5 Decide 30 in. by 57 in. Correct
Laya 30 in. by 57 in. Correct
CalDec v1 30 ft by 53 ft Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "2353",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "d85181e8b9af4750170eb6233102f11d3bdbc7aa69ff3311942b6571ca795736"
}

Type to search questions, variants and documentation.