RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 32

Back to Communication · All capabilities

Question ID: pub/e8d5810356fbc3a7e910
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Context: The scalpel performed a life-saving surgery Question: What does "scalpel" represents here?

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: knife
  • B: sharp cut
  • C: surgeon
  • D: None

Reference answer: surgeon (C)

Exact evaluated input as JSON
{
  "state": "Context: The scalpel performed a life-saving surgery\nQuestion: What does \"scalpel\" represents here?\n",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "knife",
    "B": "sharp cut",
    "C": "surgeon",
    "D": "None"
  }
}

Recorded model answers

Model Answer Outcome
Astra surgeon Correct
Opus 5.5 surgeon Correct
GLiDE v1.2 surgeon Correct
Decider v1.1 surgeon Correct
Jev surgeon Correct
Microsoft Decision-1 surgeon Correct
Clef knife Incorrect
Luna Decisions surgeon Correct
Clef Flash surgeon Correct
Strands Decider knife Incorrect
GLiNER2.5 Decide knife Incorrect
Laya knife Incorrect
CalDec v1 knife Incorrect

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_14.zip/task_14.jsonl",
    "record": "59",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "c2cae438a892e61742f83a1d544db48718f9bca39c6c872021bc04ec3de28473"
}

Type to search questions, variants and documentation.