RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 64

Back to Communication · All capabilities

Question ID: pub/e94b4356bcf8400696f3
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Speaker_1: They completely trusted him Speaker_2: True, They would follow him nowhere

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: Agrees
  • B: Sarcastic

Reference answer: Sarcastic (B)

Exact evaluated input as JSON
{
  "state": "Speaker_1: They completely trusted him\nSpeaker_2: True, They would follow him nowhere",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "Agrees",
    "B": "Sarcastic"
  }
}

Recorded model answers

Model Answer Outcome
Astra Sarcastic Correct
Opus 5.5 Sarcastic Correct
GLiDE v1.2 Sarcastic Correct
Decider v1.1 Sarcastic Correct
Jev Sarcastic Correct
Microsoft Decision-1 Sarcastic Correct
Clef Sarcastic Correct
Luna Decisions Sarcastic Correct
Clef Flash Sarcastic Correct
Strands Decider Agrees Incorrect
GLiNER2.5 Decide Agrees Incorrect
Laya Sarcastic Correct
CalDec v1 Agrees Incorrect

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_6.zip/task_6.jsonl",
    "record": "1655",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "3ad1974ca7c0335eb7fdd139082d90794b007597db395a38dccd69e0623810a3"
}

Type to search questions, variants and documentation.