RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 13

Back to Concepts and categorization · All capabilities

Question ID: socialiqa/470a52712bac93310388
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Quinn hit a home run to win the championship for her team.

Question

How would Quinn feel afterwards?

Answer choices

  • A: Disheartened
  • B: Thrilled and fulfilled
  • C: Sad and discontent

Reference answer: Thrilled and fulfilled (B)

Exact evaluated input as JSON
{
  "state": "Quinn hit a home run to win the championship for her team.",
  "question": "How would Quinn feel afterwards?",
  "choices": {
    "A": "Disheartened",
    "B": "Thrilled and fulfilled",
    "C": "Sad and discontent"
  }
}

Recorded model answers

Model Answer Outcome
Astra Thrilled and fulfilled Correct
Opus 5.5 Thrilled and fulfilled Correct
GLiDE v1.2 Thrilled and fulfilled Correct
Decider v1.1 Thrilled and fulfilled Correct
Jev Thrilled and fulfilled Correct
Microsoft Decision-1 Thrilled and fulfilled Correct
Clef Thrilled and fulfilled Correct
Luna Decisions Thrilled and fulfilled Correct
Clef Flash Thrilled and fulfilled Correct
Strands Decider Thrilled and fulfilled Correct
GLiNER2.5 Decide Thrilled and fulfilled Correct
Laya Thrilled and fulfilled Correct
CalDec v1 Thrilled and fulfilled Correct

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "16342",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "4d143261594af15def9e319dda66cd87d4088f016c2809122f0fc6d2e8b01546"
}

Type to search questions, variants and documentation.