RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 52

Back to Social cognition · All capabilities

Question ID: socialiqa/97bd80503bfec4f09f9f
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Quinn stuck her head in the sand at the beach to make her brother laugh.

Question

What will Quinn want to do next?

Answer choices

  • A: start crying
  • B: hit her brother
  • C: brush off the sand

Reference answer: brush off the sand (C)

Exact evaluated input as JSON
{
  "state": "Quinn stuck her head in the sand at the beach to make her brother laugh.",
  "question": "What will Quinn want to do next?",
  "choices": {
    "A": "start crying",
    "B": "hit her brother",
    "C": "brush off the sand"
  }
}

Recorded model answers

Model Answer Outcome
Astra brush off the sand Correct
Opus 5.5 brush off the sand Correct
GLiDE v1.2 brush off the sand Correct
Decider v1.1 brush off the sand Correct
Jev brush off the sand Correct
Microsoft Decision-1 brush off the sand Correct
Clef brush off the sand Correct
Luna Decisions brush off the sand Correct
Clef Flash brush off the sand Correct
Strands Decider brush off the sand Correct
GLiNER2.5 Decide hit her brother Incorrect
Laya hit her brother Incorrect
CalDec v1 brush off the sand Correct

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "20667",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "f7d08311eb11942f5b0d070143bf183de402cd27555056740546083a68d2e8f0"
}

Type to search questions, variants and documentation.