Retrieval · question 40
Back to Retrieval · All capabilities
Question ID: babi/cc3432e648d116c87be2
Source: bAbI
Input format: adapted choices
Explore this question’s altered versions
Context
Sandra moved to the bedroom. Daniel journeyed to the garden. Mary moved to the bedroom. Sandra went back to the hallway. John went to the office. Sandra went back to the bathroom.
Question
Where is Daniel?
Answer choices
- A: bathroom
- B: garden
- C: kitchen
- D: office
- E: bedroom
- F: hallway
Reference answer: garden (B)
Exact evaluated input as JSON
{
"state": "Sandra moved to the bedroom.\nDaniel journeyed to the garden.\nMary moved to the bedroom.\nSandra went back to the hallway.\nJohn went to the office.\nSandra went back to the bathroom.",
"question": "Where is Daniel? ",
"choices": {
"A": "bathroom",
"B": "garden",
"C": "kitchen",
"D": "office",
"E": "bedroom",
"F": "hallway"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | garden | Correct |
| Opus 5.5 | garden | Correct |
| GLiDE v1.2 | garden | Correct |
| Decider v1.1 | garden | Correct |
| Jev | garden | Correct |
| Microsoft Decision-1 | garden | Correct |
| Clef | garden | Correct |
| Luna Decisions | garden | Correct |
| Clef Flash | garden | Correct |
| Strands Decider | garden | Correct |
| GLiNER2.5 Decide | bathroom | Incorrect |
| Laya | bathroom | Incorrect |
| CalDec v1 | bathroom | Incorrect |
Source and sharing
Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
"record": "2168",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "28cfacf0ff8af2f36ed71c3a365e599a59af4a2445f0bec8968f49a90b27438c"
}