RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 4

Back to Concepts and categorization · All capabilities

Question ID: emowoz/951d42c35c1050050170
Source: EmoWOZ
Input format: native choices

Explore this question’s altered versions

Context

User: I'm looking for a place. I think it's called like Bridge House or Bridge Guesthouse or something. Assistant: You are looking for the bridge guest house User: Could you book it for 3 people and 4 nights starting from thursday? Assistant: Booking was unsuccessful. User: Okay, book it for 4 nights starting Saturday. Assistant: Booking was successful. Reference number is : EPWUAWM5. User: Thank you, that's all I need. Good bye.

Question

What emotion does the final user message express, taking the conversation into account?

Answer choices

  • 1: Fearful
  • 4: Abusive
  • 3: Apologetic
  • 6: Satisfied
  • 0: Neutral
  • 5: Excited
  • 2: Dissatisfied

Reference answer: Satisfied (6)

Exact evaluated input as JSON
{
  "state": "User: I'm looking for a place. I think it's called like Bridge House or Bridge Guesthouse or something.\nAssistant: You are looking for the bridge guest house\nUser: Could you book it for 3 people and 4 nights starting from thursday?\nAssistant: Booking was unsuccessful.\nUser: Okay, book it for 4 nights starting Saturday.\nAssistant: Booking was successful.\nReference number is : EPWUAWM5.\nUser: Thank you, that's all I need. Good bye.",
  "question": "What emotion does the final user message express, taking the conversation into account?",
  "choices": {
    "1": "Fearful",
    "4": "Abusive",
    "3": "Apologetic",
    "6": "Satisfied",
    "0": "Neutral",
    "5": "Excited",
    "2": "Dissatisfied"
  }
}

Recorded model answers

Model Answer Outcome
Astra Satisfied Correct
Opus 5.5 Satisfied Correct
GLiDE v1.2 Satisfied Correct
Decider v1.1 Satisfied Correct
Jev Satisfied Correct
Microsoft Decision-1 Neutral Incorrect
Clef Satisfied Correct
Luna Decisions Satisfied Correct
Clef Flash Satisfied Correct
Strands Decider Satisfied Correct
GLiNER2.5 Decide Neutral Incorrect
Laya Satisfied Correct
CalDec v1 Satisfied Correct

Source and sharing

Attribution: Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk, Milica Gasic.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/prepared/emowoz.jsonl",
    "record": "16425",
    "kind": "prepared_normalizer_record",
    "native_id": "emowoz/SNG02096.json/6/baseline"
  },
  "source_revision": "6506504",
  "input_hash": "3326f5bd283a0787c66020c6dbdc64acff726b528987d94a94526634039b7ae7"
}

Type to search questions, variants and documentation.