RaindropA practical guide to choosing decision modelsGitHub ↗

Concepts and categorization · question 24

Back to Concepts and categorization · All capabilities

Question ID: socialiqa/2e44383838889746f4be
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Kai gave them a huge hug when they were feeling down.

Question

How would you describe Kai?

Answer choices

  • A: uncaring
  • B: affectionate
  • C: rude

Reference answer: affectionate (B)

Exact evaluated input as JSON
{
  "state": "Kai gave them a huge hug when they were feeling down.",
  "question": "How would you describe Kai?",
  "choices": {
    "A": "uncaring",
    "B": "affectionate",
    "C": "rude"
  }
}

Recorded model answers

Model Answer Outcome
Astra affectionate Correct
Opus 5.5 affectionate Correct
GLiDE v1.2 affectionate Correct
Decider v1.1 affectionate Correct
Jev affectionate Correct
Microsoft Decision-1 affectionate Correct
Clef affectionate Correct
Luna Decisions affectionate Correct
Clef Flash affectionate Correct
Strands Decider affectionate Correct
GLiNER2.5 Decide affectionate Correct
Laya affectionate Correct
CalDec v1 affectionate Correct

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
    "record": "14329",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "764f1d885de670359f894927e05a19ccf139582d3390ed87591ba2df6e3837a6"
}

Type to search questions, variants and documentation.