RaindropA practical guide to choosing decision modelsGitHub ↗

Option order · Move the first answer choice to the end variant

Original question · Other versions of this question · All variants

This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.

Changed fields: answer choices or their order.

How the edits were constructed

Context

Conversation: Speaker_1: was Mila in the garden Speaker_2: she was in the garden or the hallway Speaker_1: where are the pineapples Speaker_2: Aria said there is a green treasure chest in the garden Speaker_1: did Mila place the pineapples there Speaker_2: she placed them there and travelled to the hallway Speaker_1: are some of the pineapples there Speaker_2: two pineapples are in the garden Speaker_1: were you there Speaker_2: I was not there Speaker_1: then where Speaker_2: I journeyed to the hallway and the back_yard Speaker_1: where can I find the celery Speaker_2: there is a red crate in the garden Speaker_1: did Aria place the celery there Speaker_2: yes Speaker_1: where are the oranges Speaker_2: the oranges are in the back_yard or the hallway Speaker_1: how about the beans Speaker_2: they are in the back_yard or the workshop Question: did Aria put the celery in the garden?

Question

Choose the interpretation or answer requested by the text.

Answer choices

  • B. no
  • A. yes

Reference answer: A.

Exact evaluated input as JSON
{
  "state": "Conversation:\nSpeaker_1: was Mila in the garden\nSpeaker_2: she was in the garden or the hallway\nSpeaker_1: where are the pineapples\nSpeaker_2: Aria said there is a green treasure chest in the garden\nSpeaker_1: did Mila place the pineapples there\nSpeaker_2: she placed them there and travelled to the hallway\nSpeaker_1: are some of the pineapples there\nSpeaker_2: two pineapples are in the garden\nSpeaker_1: were you there\nSpeaker_2: I was not there\nSpeaker_1: then where\nSpeaker_2: I journeyed to the hallway and the back_yard\nSpeaker_1: where can I find the celery\nSpeaker_2: there is a red crate in the garden\nSpeaker_1: did Aria place the celery there\nSpeaker_2: yes\nSpeaker_1: where are the oranges\nSpeaker_2: the oranges are in the back_yard or the hallway\nSpeaker_1: how about the beans\nSpeaker_2: they are in the back_yard or the workshop\nQuestion: did Aria put the celery in the garden?\n",
  "question": "Choose the interpretation or answer requested by the text.",
  "choices": {
    "B": "no",
    "A": "yes"
  }
}

Recorded model answers

The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.

Model Original answer Variant answer Change
GLiDE v1.2 A — yes A — yes Same answer
Decider v1.1 A — yes A — yes Same answer
Jev A — yes A — yes Same answer
Microsoft Decision-1 A — yes A — yes Same answer
Clef A — yes A — yes Same answer
Luna Decisions A — yes A — yes Same answer
Clef Flash A — yes A — yes Same answer
Strands Decider A — yes A — yes Same answer
GLiNER2.5 Decide A — yes A — yes Same answer
Laya A — yes A — yes Same answer
CalDec v1 A — yes A — yes Same answer

Clef and Clef Flash sort answer choices in their published input-processing code. Their choice-order results describe that preprocessing.

Identifiers and recorded conditions

Variant ID: pub/107251d5eb85b300c265/option_order/rotate_1

Input fingerprint: f78389344688fa9b0745d150da97187d7b60f6a0497f612e5e64327ab34d8c92

Change Recorded version
Option order Move the first answer choice to the end

Source attribution and dataset terms are linked from the original question.

Type to search questions, variants and documentation.