RaindropA practical guide to choosing decision modelsGitHub ↗

Rule application · question 65

Back to Rule application · All capabilities

Question ID: cruxeval/c506a16c048b3e775d6a
Source: CRUXEval
Input format: adapted choices

Explore this question’s altered versions

Context

def f(d):
    keys = []
    for k in d:
        keys.append('%s => %s' % (k, d[k]))
    return keys

Question

What does f({'-4':'4','1':'2','-':'-3'}) return?

Answer choices

  • A: ['-4 => 4', '1 => 2', '- => -3']
  • B: ['-4 => 4', '1 => 2', '= => -3']
  • C: ['-4 => 4', '1 => 2', '3- => -']
  • D: ['-4 => 4', '2 => 2', '- => -3']

Reference answer: ['-4 => 4', '1 => 2', '- => -3'] (A)

Exact evaluated input as JSON
{
  "state": "def f(d):\n    keys = []\n    for k in d:\n        keys.append('%s => %s' % (k, d[k]))\n    return keys",
  "question": "What does f({'-4':'4','1':'2','-':'-3'}) return?",
  "choices": {
    "A": "['-4 => 4', '1 => 2', '- => -3']",
    "B": "['-4 => 4', '1 => 2', '= => -3']",
    "C": "['-4 => 4', '1 => 2', '3- => -']",
    "D": "['-4 => 4', '2 => 2', '- => -3']"
  }
}

Recorded model answers

Model Answer Outcome
Astra ['-4 => 4', '1 => 2', '- => -3'] Correct
Opus 5.5 ['-4 => 4', '1 => 2', '- => -3'] Correct
GLiDE v1.2 ['-4 => 4', '1 => 2', '- => -3'] Correct
Decider v1.1 ['-4 => 4', '1 => 2', '- => -3'] Correct
Jev ['-4 => 4', '1 => 2', '- => -3'] Correct
Microsoft Decision-1 ['-4 => 4', '1 => 2', '- => -3'] Correct
Clef ['-4 => 4', '1 => 2', '- => -3'] Correct
Luna Decisions ['-4 => 4', '1 => 2', '- => -3'] Correct
Clef Flash ['-4 => 4', '1 => 2', '- => -3'] Correct
Strands Decider ['-4 => 4', '1 => 2', '- => -3'] Correct
GLiNER2.5 Decide ['-4 => 4', '2 => 2', '- => -3'] Incorrect
Laya ['-4 => 4', '1 => 2', '- => -3'] Correct
CalDec v1 ['-4 => 4', '1 => 2', '- => -3'] Correct

Source and sharing

Attribution: Meta; CRUXEval authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/cruxeval.jsonl",
    "record": "493",
    "kind": "source_file_record"
  },
  "source_revision": "190faf16d175b5847b0af05d937872b1fb395942",
  "input_hash": "9aa281a1f9f66306db88b8933cf5315bce9019e85375d8c5ddfb44d7cc34a73f"
}

Type to search questions, variants and documentation.