RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 32

Back to Social cognition · All capabilities

Question ID: socialiqa/e855db00e65efa9913e8
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Lee listened to Kendall's talk and took a lot of notes.

Question

Why did Lee do this?

Answer choices

  • A: understand Kendall's words
  • B: ruin Kai's reputation
  • C: tell lies about Kendall

Reference answer: understand Kendall's words (A)

Exact evaluated input as JSON
{
  "state": "Lee listened to Kendall's talk and took a lot of notes.",
  "question": "Why did Lee do this?",
  "choices": {
    "A": "understand Kendall's words",
    "B": "ruin Kai's reputation",
    "C": "tell lies about Kendall"
  }
}

Recorded model answers

Model Answer Outcome
Astra understand Kendall's words Correct
Opus 5.5 understand Kendall's words Correct
GLiDE v1.2 understand Kendall's words Correct
Decider v1.1 understand Kendall's words Correct
Jev understand Kendall's words Correct
Microsoft Decision-1 understand Kendall's words Correct
Clef understand Kendall's words Correct
Luna Decisions understand Kendall's words Correct
Clef Flash understand Kendall's words Correct
Strands Decider understand Kendall's words Correct
GLiNER2.5 Decide understand Kendall's words Correct
Laya understand Kendall's words Correct
CalDec v1 understand Kendall's words Correct

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "33265",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "45f2a47f9058faeee4cbc33b93436d0ee7b9138318beca2e4f6ec18d5b03f3b7"
}

Type to search questions, variants and documentation.