RaindropA practical guide to choosing decision modelsGitHub ↗

Numerical cognition · question 32

Back to Numerical cognition · All capabilities

Question ID: mmlu/b94c0a08436343bbf655
Source: MMLU
Input format: native choices

Explore this question’s altered versions

Question

Two computers communicate with each other by sending data packets across a local area network. The size of these packets is 1,000 bytes. The network has the capacity to carry 1,000 packets per second. The CPU time required to execute the network protocol to send one packet is 10 milliseconds. The maximum rate at which one computer can send data to another is approximately

Answer choices

  • A: 10,000 bytes/second
  • B: 25,000 bytes/ second
  • C: 100,000 bytes/ second
  • D: 500,000 bytes/ second

Reference answer: 100,000 bytes/ second (C)

Exact evaluated input as JSON
{
  "state": "",
  "question": "Two computers communicate with each other by sending data packets across a local area network. The size of these packets is 1,000 bytes. The network has the capacity to carry 1,000 packets per second. The CPU time required to execute the network protocol to send one packet is 10 milliseconds. The maximum rate at which one computer can send data to another is approximately",
  "choices": {
    "A": "10,000 bytes/second",
    "B": "25,000 bytes/ second",
    "C": "100,000 bytes/ second",
    "D": "500,000 bytes/ second"
  }
}

Recorded model answers

Model Answer Outcome
Astra 100,000 bytes/ second Correct
Opus 5.5 100,000 bytes/ second Correct
GLiDE v1.2 500,000 bytes/ second Incorrect
Decider v1.1 100,000 bytes/ second Correct
Jev 100,000 bytes/ second Correct
Microsoft Decision-1 10,000 bytes/second Incorrect
Clef 100,000 bytes/ second Correct
Luna Decisions 100,000 bytes/ second Correct
Clef Flash 100,000 bytes/ second Correct
Strands Decider 100,000 bytes/ second Correct
GLiNER2.5 Decide 10,000 bytes/second Incorrect
Laya 10,000 bytes/second Incorrect
CalDec v1 10,000 bytes/second Incorrect

Source and sharing

Attribution: Dan Hendrycks et al.; MMLU authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "all/test-00000-of-00001.parquet",
    "record": "1069",
    "kind": "source_file_record"
  },
  "source_revision": "c30699e8356da336a370243923dbaf21066bb9fe",
  "input_hash": "cd13228d48de401f214b244900346c5bbf62ef35486a32785ac9fe98c64c7f52"
}

Type to search questions, variants and documentation.