RaindropA practical guide to choosing decision modelsGitHub ↗

Physical cognition · question 12

Back to Physical cognition · All capabilities

Question ID: mmlu/fe496c5645f5c78aad17
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

A toy car and a toy truck collide. If the toy truck’s mass is double the toy car’s mass, then, compared to the acceleration of the truck, the acceleration of the car during the collision will be

Answer choices

  • A: double the magnitude and in the same direction
  • B: double the magnitude and in the opposite direction
  • C: half the magnitude and in the same direction
  • D: half the magnitude and in the opposite direction

Reference answer: double the magnitude and in the opposite direction (B)

Exact evaluated input as JSON
{
  "state": "",
  "question": "A toy car and a toy truck collide. If the toy truck’s mass is double the toy car’s mass, then, compared to the acceleration of the truck, the acceleration of the car during the collision will be",
  "choices": {
    "A": "double the magnitude and in the same direction",
    "B": "double the magnitude and in the opposite direction",
    "C": "half the magnitude and in the same direction",
    "D": "half the magnitude and in the opposite direction"
  }
}

Recorded model answers

Model Answer Outcome
Astra double the magnitude and in the opposite direction Correct
Opus 5.5 double the magnitude and in the opposite direction Correct
GLiDE v1.2 double the magnitude and in the opposite direction Correct
Decider v1.1 double the magnitude and in the opposite direction Correct
Jev double the magnitude and in the opposite direction Correct
Microsoft Decision-1 double the magnitude and in the opposite direction Correct
Clef double the magnitude and in the opposite direction Correct
Luna Decisions double the magnitude and in the opposite direction Correct
Clef Flash double the magnitude and in the opposite direction Correct
Strands Decider half the magnitude and in the same direction Incorrect
GLiNER2.5 Decide double the magnitude and in the opposite direction Correct
Laya double the magnitude and in the same direction Incorrect
CalDec v1 double the magnitude and in the same direction Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "4813",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "d560e0b63d960089bbf3bb48ef9d42aed485ab77695816e2df79d9b16b114bc8"
}

Type to search questions, variants and documentation.