RaindropA practical guide to choosing decision modelsGitHub ↗

Rule application · question 17

Back to Rule application · All capabilities

Question ID: cruxeval/c84ac060fadf2aac6420
Source: CRUXEval
Input format: adapted choices

Explore this question’s altered versions

Context

def f(text, tabstop):
    text = text.replace('\n', '_____')
    text = text.replace('\t', tabstop * ' ')
    text = text.replace('_____', '\n')
    return text

Question

What does f("odes\tcode\twell", 2) return?

Answer choices

  • A: 'odes code owell'
  • B: 'odes c de well'
  • C: 'odes wcode well'
  • D: 'odes code well'

Reference answer: 'odes code well' (D)

Exact evaluated input as JSON
{
  "state": "def f(text, tabstop):\n    text = text.replace('\\n', '_____')\n    text = text.replace('\\t', tabstop * ' ')\n    text = text.replace('_____', '\\n')\n    return text",
  "question": "What does f(\"odes\\tcode\\twell\", 2) return?",
  "choices": {
    "A": "'odes  code owell'",
    "B": "'odes  c de  well'",
    "C": "'odes wcode  well'",
    "D": "'odes  code  well'"
  }
}

Recorded model answers

Model Answer Outcome
Astra 'odes code well' Correct
Opus 5.5 'odes code well' Correct
GLiDE v1.2 'odes code well' Correct
Decider v1.1 'odes code well' Correct
Jev 'odes code well' Correct
Microsoft Decision-1 'odes code well' Correct
Clef 'odes code well' Correct
Luna Decisions 'odes code well' Correct
Clef Flash 'odes code well' Correct
Strands Decider 'odes code well' Correct
GLiNER2.5 Decide 'odes code owell' Incorrect
Laya 'odes code owell' Incorrect
CalDec v1 'odes code owell' Incorrect

Source and sharing

Attribution: Meta; CRUXEval authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/cruxeval.jsonl",
    "record": "555",
    "kind": "source_file_record"
  },
  "source_revision": "190faf16d175b5847b0af05d937872b1fb395942",
  "input_hash": "77778b1140eb032577e61f26a9fc96e64404a64cce82cc49b604c842e0736869"
}

Type to search questions, variants and documentation.