RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 7

Back to Social cognition · All capabilities

Question ID: socialiqa/2c751698a78513132958
Source: Social IQa
Input format: native choices

Explore this question’s altered versions

Context

Austin got extra help with the homework from his tutor since he was falling behind in class.

Question

What will the tutor want to do next?

Answer choices

  • A: do their own homework on time
  • B: review homework with Austin
  • C: realize he was doing poorly in class

Reference answer: review homework with Austin (B)

Exact evaluated input as JSON
{
  "state": "Austin got extra help with the homework from his tutor since he was falling behind in class.",
  "question": "What will the tutor want to do next?",
  "choices": {
    "A": "do their own homework on time",
    "B": "review homework with Austin",
    "C": "realize he was doing poorly in class"
  }
}

Recorded model answers

Model Answer Outcome
Astra review homework with Austin Correct
Opus 5.5 review homework with Austin Correct
GLiDE v1.2 review homework with Austin Correct
Decider v1.1 review homework with Austin Correct
Jev review homework with Austin Correct
Microsoft Decision-1 review homework with Austin Correct
Clef review homework with Austin Correct
Luna Decisions review homework with Austin Correct
Clef Flash review homework with Austin Correct
Strands Decider review homework with Austin Correct
GLiNER2.5 Decide realize he was doing poorly in class Incorrect
Laya realize he was doing poorly in class Incorrect
CalDec v1 review homework with Austin Correct

Source and sharing

Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/socialiqa.zip/socialiqa-train-dev/dev.jsonl",
    "record": "1284",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "0c3269201de2d99fc4301055e7ca857faf2eb92ffd85b4ef83504290c8d4b82f"
}

Type to search questions, variants and documentation.