RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 50

Back to Communication · All capabilities

Question ID: pub/d2c8a99d1340e475aa35
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Context: The press is always hungry for a story Question: what does "hungry" refer to?

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: starving
  • B: Strong desire
  • C: empty
  • D: none

Reference answer: Strong desire (B)

Exact evaluated input as JSON
{
  "state": "Context: The press is always hungry for a story\nQuestion: what does \"hungry\" refer to?\n",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "starving",
    "B": "Strong desire",
    "C": "empty",
    "D": "none"
  }
}

Recorded model answers

Model Answer Outcome
Astra Strong desire Correct
Opus 5.5 Strong desire Correct
GLiDE v1.2 Strong desire Correct
Decider v1.1 Strong desire Correct
Jev Strong desire Correct
Microsoft Decision-1 Strong desire Correct
Clef Strong desire Correct
Luna Decisions Strong desire Correct
Clef Flash Strong desire Correct
Strands Decider Strong desire Correct
GLiNER2.5 Decide starving Incorrect
Laya Strong desire Correct
CalDec v1 Strong desire Correct

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_14.zip/task_14.jsonl",
    "record": "412",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "844b93d93dac942f8ec01b91f9a3222fe1c37530a60ebd6aa94c0a5f7c2e37e5"
}

Type to search questions, variants and documentation.