RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 56

Back to Social cognition · All capabilities

Question ID: socialiqa/de2b2c2fa75e5a5f3f12
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Kendall ate a hamburger from the local diner and later felt sick to their stomach.

Question

What will Kendall want to do next?

Answer choices

  • A: eat another hamburger
  • B: complain to the diner's owner
  • C: order a burger

Reference answer: complain to the diner's owner (B)

Exact evaluated input as JSON
{
  "state": "Kendall ate a hamburger from the local diner and later felt sick to their stomach.",
  "question": "What will Kendall want to do next?",
  "choices": {
    "A": "eat another hamburger",
    "B": "complain to the diner's owner",
    "C": "order a burger"
  }
}

Recorded model answers

Model Answer Outcome
Astra complain to the diner's owner Correct
Opus 5.5 complain to the diner's owner Correct
GLiDE v1.2 complain to the diner's owner Correct
Decider v1.1 complain to the diner's owner Correct
Jev complain to the diner's owner Correct
Microsoft Decision-1 complain to the diner's owner Correct
Clef complain to the diner's owner Correct
Luna Decisions complain to the diner's owner Correct
Clef Flash complain to the diner's owner Correct
Strands Decider complain to the diner's owner Correct
GLiNER2.5 Decide complain to the diner's owner Correct
Laya eat another hamburger Incorrect
CalDec v1 eat another hamburger Incorrect

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "30668",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "43711c481a38f3cb4d13c04fb1e43e9270bb58e5cff681891360b9ba94034e3e"
}

Type to search questions, variants and documentation.