RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 42

Back to Communication · All capabilities

Question ID: pub/954cafe1581f8d2f0f2f
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Speaker_1: The seas on the voyage were a rock quarry Speaker_2: The seas were calm

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: Agrees
  • B: Disagrees

Reference answer: Disagrees (B)

Exact evaluated input as JSON
{
  "state": "Speaker_1: The seas on the voyage were a rock quarry\nSpeaker_2: The seas were calm",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "Agrees",
    "B": "Disagrees"
  }
}

Recorded model answers

Model Answer Outcome
Astra Disagrees Correct
Opus 5.5 Disagrees Correct
GLiDE v1.2 Disagrees Correct
Decider v1.1 Disagrees Correct
Jev Disagrees Correct
Microsoft Decision-1 Disagrees Correct
Clef Disagrees Correct
Luna Decisions Disagrees Correct
Clef Flash Disagrees Correct
Strands Decider Disagrees Correct
GLiNER2.5 Decide Disagrees Correct
Laya Disagrees Correct
CalDec v1 Disagrees Correct

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_5.zip/task_5.jsonl",
    "record": "26",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "34b84787df3d85530be09f2ed438718d561a64e40f45792850926115b1c1c606"
}

Type to search questions, variants and documentation.