RaindropA practical guide to choosing decision modelsGitHub ↗

Communication · question 16

Back to Communication · All capabilities

Question ID: pub/fac5c2c6ba4042cb48ed
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Context: X wants to know what activities Y likes to do during weekends. Question: Do you like to take some time for yourself on the weekends? Response: I spend 15 minutes a day meditating.

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: Direct answer
  • B: Indirect answer

Reference answer: Indirect answer (B)

Exact evaluated input as JSON
{
  "state": "Context: X wants to know what activities Y likes to do during weekends.\nQuestion: Do you like to take some time for yourself on the weekends?\nResponse: I spend 15 minutes a day meditating.\n",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "Direct answer",
    "B": "Indirect answer"
  }
}

Recorded model answers

Model Answer Outcome
Astra Indirect answer Correct
Opus 5.5 Indirect answer Correct
GLiDE v1.2 Indirect answer Correct
Decider v1.1 Indirect answer Correct
Jev Indirect answer Correct
Microsoft Decision-1 Indirect answer Correct
Clef Indirect answer Correct
Luna Decisions Indirect answer Correct
Clef Flash Direct answer Incorrect
Strands Decider Indirect answer Correct
GLiNER2.5 Decide Indirect answer Correct
Laya Indirect answer Correct
CalDec v1 Indirect answer Correct

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_1.zip/task_1.jsonl",
    "record": "2499",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "2bcb282501d6d5f711da0bb930eaac8a207e8125a553df5bdb66ecfcde33b8b7"
}

Type to search questions, variants and documentation.