Social cognition · question 44
Back to Social cognition · All capabilities
Question ID: socialiqa/a0337f76dfd022f2f0aa
Source: Social IQa
Input format: native choices
Explore this question’s altered versions
Context
Skylar was grateful for Robin's letter of recommendation and eager to apply for the position. Furthermore, Robin knew Skylar personally.
Question
What will Skylar want to do next?
Answer choices
- A: thank Robin for the help
- B: a bad idea
- C: avoid contact with Robin
Reference answer: thank Robin for the help (A)
Exact evaluated input as JSON
{
"state": "Skylar was grateful for Robin's letter of recommendation and eager to apply for the position. Furthermore, Robin knew Skylar personally.",
"question": "What will Skylar want to do next?",
"choices": {
"A": "thank Robin for the help",
"B": "a bad idea",
"C": "avoid contact with Robin"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | thank Robin for the help | Correct |
| Opus 5.5 | thank Robin for the help | Correct |
| GLiDE v1.2 | thank Robin for the help | Correct |
| Decider v1.1 | thank Robin for the help | Correct |
| Jev | thank Robin for the help | Correct |
| Microsoft Decision-1 | thank Robin for the help | Correct |
| Clef | thank Robin for the help | Correct |
| Luna Decisions | thank Robin for the help | Correct |
| Clef Flash | thank Robin for the help | Correct |
| Strands Decider | thank Robin for the help | Correct |
| GLiNER2.5 Decide | thank Robin for the help | Correct |
| Laya | thank Robin for the help | Correct |
| CalDec v1 | thank Robin for the help | Correct |
Source and sharing
Attribution: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/socialiqa.zip/socialiqa-train-dev/train.jsonl",
"record": "31104",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "139d51a42a32f39e21ccae950ff97b628062de5225a4fcd5c80efe6b46c150b1"
}