RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 28

Back to Communication · All capabilities

Question ID: pub/77068aa5ee856d90a61f
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Speaker_1: They do not waste food at all Speaker_2: Yes, They waste as much food as a king's spoiled prince

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: Agrees
  • B: Sarcastic

Reference answer: Sarcastic (B)

Exact evaluated input as JSON
{
  "state": "Speaker_1: They do not waste food at all\nSpeaker_2: Yes, They waste as much food as a king's spoiled prince",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "Agrees",
    "B": "Sarcastic"
  }
}

Recorded model answers

Model Answer Outcome
Astra Sarcastic Correct
Opus 5.5 Sarcastic Correct
GLiDE v1.2 Sarcastic Correct
Decider v1.1 Sarcastic Correct
Jev Sarcastic Correct
Microsoft Decision-1 Sarcastic Correct
Clef Sarcastic Correct
Luna Decisions Sarcastic Correct
Clef Flash Sarcastic Correct
Strands Decider Sarcastic Correct
GLiNER2.5 Decide Sarcastic Correct
Laya Sarcastic Correct
CalDec v1 Agrees Incorrect

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_6.zip/task_6.jsonl",
    "record": "422",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "d32655fda6bccaabba439ed8925f2b29cbcf25f65987c011685bc8c71f961a7a"
}

Type to search questions, variants and documentation.