RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 40

Back to Concepts and categorization · All capabilities

Question ID: emowoz/6655f90bafecb88cc636
Source: EmoWOZ
Input format: native choices

Explore this question’s altered versions

Context

User: I need a train departing from Cambridge, leaving after 21:30. Assistant: What time were you looking to arrive and what is your destination? User: I need to go to Peterborough on Tuesday please Assistant: The TR7663 departs at 21:34 and arrives in Peterborough at 22:24. Shall I book this for you? User: Yes, for 7 people please. I need the reference number. Assistant: Booking was successful, the total fee is 115.5 GBP payable at the station. Reference number is : NBX6ZT2O. User: I'm also looking for a restaurant in the south on the expensive side. Assistant: I have 5 results for expensive restaurants in the south. Are you in the mood for Chinese, Indian, Italian or Mexican food? User: We are looking for indian food. Assistant: The Taj Tandoori is an expensive restaurant in the south. It is located at 64 Cherry Hinton Road. Would you like me to book it? User: Yes, for 7 people at 10:30 on Tuesday. Assistant: Unfortunately the restaurant is full at that time. Would you like to try another time or day? User: How about 9:30? Assistant: I'm sorry, they don't offer that time. Would you like to try a later time? User: They have no 9:30? Assistant: They have no bookings available Tuesday morning. Would you like to try afternoon or another restaurant? User: Let's try again. An expensive Indian restaurant in the south - Tuesday, 9:30, 7 people.

Question

What emotion does the final user message express, taking the conversation into account?

Answer choices

  • 4: Abusive
  • 5: Excited
  • 1: Fearful
  • 6: Satisfied
  • 0: Neutral
  • 2: Dissatisfied
  • 3: Apologetic

Reference answer: Dissatisfied (2)

Exact evaluated input as JSON
{
  "state": "User: I need a train departing from Cambridge, leaving after 21:30. \nAssistant: What time were you looking to arrive and what is your destination?\nUser: I need to go to Peterborough on Tuesday please\nAssistant: The TR7663 departs at 21:34 and arrives in Peterborough at 22:24. Shall I book this for you?\nUser: Yes, for 7 people please.  I need the reference number.\nAssistant: Booking was successful, the total fee is 115.5 GBP payable at the station. Reference number is : NBX6ZT2O.\nUser: I'm also looking for a restaurant in the south on the expensive side. \nAssistant: I have 5 results for expensive restaurants in the south. Are you in the mood for Chinese, Indian, Italian or Mexican food?\nUser: We are looking for indian food.\nAssistant: The Taj Tandoori is an expensive restaurant in the south.  It is located at 64 Cherry Hinton Road.  Would you like me to book it?\nUser: Yes, for 7 people at 10:30 on Tuesday.\nAssistant: Unfortunately the restaurant is full at that time. Would you like to try another time or day?\nUser: How about 9:30? \nAssistant: I'm sorry, they don't offer that time. Would you like to try a later time?\nUser: They have no 9:30?\nAssistant: They have no bookings available Tuesday morning. Would you like to try afternoon or another restaurant?\nUser: Let's try again. An expensive Indian restaurant in the south - Tuesday, 9:30, 7 people.",
  "question": "What emotion does the final user message express, taking the conversation into account?",
  "choices": {
    "4": "Abusive",
    "5": "Excited",
    "1": "Fearful",
    "6": "Satisfied",
    "0": "Neutral",
    "2": "Dissatisfied",
    "3": "Apologetic"
  }
}

Recorded model answers

Model Answer Outcome
Astra Dissatisfied Correct
Opus 5.5 Dissatisfied Correct
GLiDE v1.2 Neutral Incorrect
Decider v1.1 Neutral Incorrect
Jev Dissatisfied Correct
Microsoft Decision-1 Neutral Incorrect
Clef Neutral Incorrect
Luna Decisions Dissatisfied Correct
Clef Flash Neutral Incorrect
Strands Decider Neutral Incorrect
GLiNER2.5 Decide Dissatisfied Correct
Laya Satisfied Incorrect
CalDec v1 Satisfied Incorrect

Source and sharing

Attribution: Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk, Milica Gasic.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/prepared/emowoz.jsonl",
    "record": "10190",
    "kind": "prepared_normalizer_record",
    "native_id": "emowoz/MUL0370.json/16/baseline"
  },
  "source_revision": "6506504",
  "input_hash": "9905fbdb73f13d669c8cc40988afb3d9b6169f2a058f7d951815a6fe662a6763"
}

Type to search questions, variants and documentation.