Social cognition · question 7
Back to Social cognition · All capabilities
Question ID: socialiqa/2c751698a78513132958
Source: Social IQa
Input format: native choices
Explore this question’s altered versions
Context
Austin got extra help with the homework from his tutor since he was falling behind in class.
Question
What will the tutor want to do next?
Answer choices
- A: do their own homework on time
- B: review homework with Austin
- C: realize he was doing poorly in class
Reference answer: review homework with Austin (B)
Exact evaluated input as JSON
{
"state": "Austin got extra help with the homework from his tutor since he was falling behind in class.",
"question": "What will the tutor want to do next?",
"choices": {
"A": "do their own homework on time",
"B": "review homework with Austin",
"C": "realize he was doing poorly in class"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | review homework with Austin | Correct |
| Opus 5.5 | review homework with Austin | Correct |
| GLiDE v1.2 | review homework with Austin | Correct |
| Decider v1.1 | review homework with Austin | Correct |
| Jev | review homework with Austin | Correct |
| Microsoft Decision-1 | review homework with Austin | Correct |
| Clef | review homework with Austin | Correct |
| Luna Decisions | review homework with Austin | Correct |
| Clef Flash | review homework with Austin | Correct |
| Strands Decider | review homework with Austin | Correct |
| GLiNER2.5 Decide | realize he was doing poorly in class | Incorrect |
| Laya | realize he was doing poorly in class | Incorrect |
| CalDec v1 | review homework with Austin | Correct |
Source and sharing
Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/socialiqa.zip/socialiqa-train-dev/dev.jsonl",
"record": "1284",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "0c3269201de2d99fc4301055e7ca857faf2eb92ffd85b4ef83504290c8d4b82f"
}