RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 16

Back to Retrieval · All capabilities

Question ID: pub/107251d5eb85b300c265
Source: PUB
Input format: native choices

Explore this question’s altered versions

Context

Conversation: Speaker_1: was Mila in the garden Speaker_2: she was in the garden or the hallway Speaker_1: where are the pineapples Speaker_2: Aria said there is a green treasure chest in the garden Speaker_1: did Mila place the pineapples there Speaker_2: she placed them there and travelled to the hallway Speaker_1: are some of the pineapples there Speaker_2: two pineapples are in the garden Speaker_1: were you there Speaker_2: I was not there Speaker_1: then where Speaker_2: I journeyed to the hallway and the back_yard Speaker_1: where can I find the celery Speaker_2: there is a red crate in the garden Speaker_1: did Aria place the celery there Speaker_2: yes Speaker_1: where are the oranges Speaker_2: the oranges are in the back_yard or the hallway Speaker_1: how about the beans Speaker_2: they are in the back_yard or the workshop Question: did Aria put the celery in the garden?

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • A: yes
  • B: no

Reference answer: yes (A)

Exact evaluated input as JSON
{
  "state": "Conversation:\nSpeaker_1: was Mila in the garden\nSpeaker_2: she was in the garden or the hallway\nSpeaker_1: where are the pineapples\nSpeaker_2: Aria said there is a green treasure chest in the garden\nSpeaker_1: did Mila place the pineapples there\nSpeaker_2: she placed them there and travelled to the hallway\nSpeaker_1: are some of the pineapples there\nSpeaker_2: two pineapples are in the garden\nSpeaker_1: were you there\nSpeaker_2: I was not there\nSpeaker_1: then where\nSpeaker_2: I journeyed to the hallway and the back_yard\nSpeaker_1: where can I find the celery\nSpeaker_2: there is a red crate in the garden\nSpeaker_1: did Aria place the celery there\nSpeaker_2: yes\nSpeaker_1: where are the oranges\nSpeaker_2: the oranges are in the back_yard or the hallway\nSpeaker_1: how about the beans\nSpeaker_2: they are in the back_yard or the workshop\nQuestion: did Aria put the celery in the garden?\n",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "A": "yes",
    "B": "no"
  }
}

Recorded model answers

Model Answer Outcome
Astra yes Correct
Opus 5.5 yes Correct
GLiDE v1.2 yes Correct
Decider v1.1 yes Correct
Jev yes Correct
Microsoft Decision-1 yes Correct
Clef yes Correct
Luna Decisions yes Correct
Clef Flash yes Correct
Strands Decider yes Correct
GLiNER2.5 Decide yes Correct
Laya yes Correct
CalDec v1 yes Correct

Source and sharing

Attribution: CFILT, IIT Bombay; PUB authors.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/task_13.zip/task_13.jsonl",
    "record": "614",
    "kind": "source_file_record"
  },
  "source_revision": "65a6a87359fe4aa5278952741d5ed7e0ecb0f0ff",
  "input_hash": "5a63ef62a2ffcaee286b8e0fb660a10b9e279f46eed89de7fb5277032a1bfacd"
}

Type to search questions, variants and documentation.