RaindropA practical guide to choosing decision modelsGitHub ↗

Evidence order · Present the evidence in a shuffled order variant

Original question · Other versions of this question · All variants

This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.

Changed fields: context.

How the edits were constructed

Context

Read the full context

«Assistant: What time were you looking to arrive and what is your destination?» «Assistant: The TR7663 departs at 21:34 and arrives in Peterborough at 22:24. Shall I book this for you?» «Assistant: Booking was successful, the total fee is 115.5 GBP payable at the station. Reference number is : NBX6ZT2O.» «Assistant: I have 5 results for expensive restaurants in the south. Are you in the mood for Chinese, Indian, Italian or Mexican food?» «Assistant: The Taj Tandoori is an expensive restaurant in the south. It is located at 64 Cherry Hinton Road. Would you like me to book it?» «Assistant: Unfortunately the restaurant is full at that time. Would you like to try another time or day?» «Assistant: I'm sorry, they don't offer that time. Would you like to try a later time?» «Assistant: They have no bookings available Tuesday morning. Would you like to try afternoon or another restaurant?» Before the Assistant asks about arrival time and destination, «User: I need a train departing from Cambridge, leaving after 21:30. » After the Assistant asks about arrival time and destination and before the Assistant asks whether to book the proposed train, «User: I need to go to Peterborough on Tuesday please» After the Assistant asks whether to book the proposed train and before the Assistant replies to the seven-person train booking request, «User: Yes, for 7 people please. I need the reference number.» After the Assistant replies to the seven-person train booking request and before the Assistant asks which restaurant cuisine is wanted, «User: I'm also looking for a restaurant in the south on the expensive side. » After the Assistant asks which restaurant cuisine is wanted and before the Assistant asks whether to book the proposed restaurant, «User: We are looking for indian food.» After the Assistant asks whether to book the proposed restaurant and before the Assistant replies to the Tuesday restaurant booking request, «User: Yes, for 7 people at 10:30 on Tuesday.» After the Assistant replies to the Tuesday restaurant booking request and before the Assistant replies to the proposed 9:30 time, «User: How about 9:30? » After the Assistant replies to the proposed 9:30 time and before the Assistant replies to the question about 9:30 availability, «User: They have no 9:30?» After the Assistant replies to the question about 9:30 availability, «User: Let's try again. An expensive Indian restaurant in the south - Tuesday, 9:30, 7 people.»

Question

What emotion does the final user message express, taking the conversation into account?

Answer choices

  • 4. Abusive
  • 5. Excited
  • 1. Fearful
  • 6. Satisfied
  • 0. Neutral
  • 2. Dissatisfied
  • 3. Apologetic

Reference answer: 2.

Exact evaluated input as JSON
{
  "state": "«Assistant: What time were you looking to arrive and what is your destination?»\n«Assistant: The TR7663 departs at 21:34 and arrives in Peterborough at 22:24. Shall I book this for you?»\n«Assistant: Booking was successful, the total fee is 115.5 GBP payable at the station. Reference number is : NBX6ZT2O.»\n«Assistant: I have 5 results for expensive restaurants in the south. Are you in the mood for Chinese, Indian, Italian or Mexican food?»\n«Assistant: The Taj Tandoori is an expensive restaurant in the south.  It is located at 64 Cherry Hinton Road.  Would you like me to book it?»\n«Assistant: Unfortunately the restaurant is full at that time. Would you like to try another time or day?»\n«Assistant: I'm sorry, they don't offer that time. Would you like to try a later time?»\n«Assistant: They have no bookings available Tuesday morning. Would you like to try afternoon or another restaurant?»\nBefore the Assistant asks about arrival time and destination,\n«User: I need a train departing from Cambridge, leaving after 21:30. »\nAfter the Assistant asks about arrival time and destination and before the Assistant asks whether to book the proposed train,\n«User: I need to go to Peterborough on Tuesday please»\nAfter the Assistant asks whether to book the proposed train and before the Assistant replies to the seven-person train booking request,\n«User: Yes, for 7 people please.  I need the reference number.»\nAfter the Assistant replies to the seven-person train booking request and before the Assistant asks which restaurant cuisine is wanted,\n«User: I'm also looking for a restaurant in the south on the expensive side. »\nAfter the Assistant asks which restaurant cuisine is wanted and before the Assistant asks whether to book the proposed restaurant,\n«User: We are looking for indian food.»\nAfter the Assistant asks whether to book the proposed restaurant and before the Assistant replies to the Tuesday restaurant booking request,\n«User: Yes, for 7 people at 10:30 on Tuesday.»\nAfter the Assistant replies to the Tuesday restaurant booking request and before the Assistant replies to the proposed 9:30 time,\n«User: How about 9:30? »\nAfter the Assistant replies to the proposed 9:30 time and before the Assistant replies to the question about 9:30 availability,\n«User: They have no 9:30?»\nAfter the Assistant replies to the question about 9:30 availability,\n«User: Let's try again. An expensive Indian restaurant in the south - Tuesday, 9:30, 7 people.»",
  "question": "What emotion does the final user message express, taking the conversation into account?",
  "choices": {
    "4": "Abusive",
    "5": "Excited",
    "1": "Fearful",
    "6": "Satisfied",
    "0": "Neutral",
    "2": "Dissatisfied",
    "3": "Apologetic"
  }
}

Recorded model answers

The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.

Model Original answer Variant answer Change
GLiDE v1.2 0 — Neutral 0 — Neutral Same answer
Decider v1.1 0 — Neutral 0 — Neutral Same answer
Jev 2 — Dissatisfied 0 — Neutral Different answer
Microsoft Decision-1 0 — Neutral 0 — Neutral Same answer
Clef 0 — Neutral 0 — Neutral Same answer
Luna Decisions 2 — Dissatisfied 0 — Neutral Different answer
Clef Flash 0 — Neutral 0 — Neutral Same answer
Strands Decider 0 — Neutral 0 — Neutral Same answer
GLiNER2.5 Decide 2 — Dissatisfied 2 — Dissatisfied Same answer
Laya 6 — Satisfied 6 — Satisfied Same answer
CalDec v1 6 — Satisfied 6 — Satisfied Same answer

Ordering comparison: coupled wording reference chronology or representation. Only the exact matched comparison sets contribute to the reported ordering contrasts.

Identifiers and recorded conditions

Variant ID: emowoz/6655f90bafecb88cc636/presentation_order/shuffled_presentation/clean-source_first_emowoz_temporal_shuffle_full_0349

Input fingerprint: 1e9a80ae827eeecd218e4048fc1bcdec3c4bc3638520fddb064b8e71b8586894

Change Recorded version
Evidence order Present the evidence in a shuffled order

Source attribution and dataset terms are linked from the original question.

Type to search questions, variants and documentation.