RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 24

Back to Social cognition · All capabilities

Question ID: socialiqa/137e86e24691a023ec85
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Addison's fish died recently and Addison wanted to buy a replacement.

Question

Why did Addison do this?

Answer choices

  • A: fill her aquarium
  • B: get a frog
  • C: discover that their fish was dead

Reference answer: fill her aquarium (A)

Exact evaluated input as JSON
{
  "state": "Addison's fish died recently and Addison wanted to buy a replacement.",
  "question": "Why did Addison do this?",
  "choices": {
    "A": "fill her aquarium",
    "B": "get a frog",
    "C": "discover that their fish was dead"
  }
}

Recorded model answers

Model Answer Outcome
Astra fill her aquarium Correct
Opus 5.5 fill her aquarium Correct
GLiDE v1.2 fill her aquarium Correct
Decider v1.1 fill her aquarium Correct
Jev fill her aquarium Correct
Microsoft Decision-1 discover that their fish was dead Incorrect
Clef fill her aquarium Correct
Luna Decisions fill her aquarium Correct
Clef Flash fill her aquarium Correct
Strands Decider discover that their fish was dead Incorrect
GLiNER2.5 Decide discover that their fish was dead Incorrect
Laya fill her aquarium Correct
CalDec v1 discover that their fish was dead Incorrect

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "25238",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "24caae91d5a7746a70a2fd2b51c06edefe0558388214280a291b0e128e45dc59"
}

Type to search questions, variants and documentation.