State tracking and updating · question 55
Back to State tracking and updating · All capabilities
Question ID: babi/59637e1b402bb9904895
Source: bAbI
Input format: adapted choices
Explore this question’s altered versions
Context
Daniel travelled to the office. Sandra went back to the office. Mary went to the office. Daniel moved to the bathroom. John went to the hallway. Sandra moved to the bedroom. Sandra travelled to the office. John went back to the garden.
Question
Where is Sandra?
Answer choices
- A: garden
- B: bedroom
- C: kitchen
- D: office
- E: hallway
- F: bathroom
Reference answer: office (D)
Exact evaluated input as JSON
{
"state": "Daniel travelled to the office.\nSandra went back to the office.\nMary went to the office.\nDaniel moved to the bathroom.\nJohn went to the hallway.\nSandra moved to the bedroom.\nSandra travelled to the office.\nJohn went back to the garden.",
"question": "Where is Sandra? ",
"choices": {
"A": "garden",
"B": "bedroom",
"C": "kitchen",
"D": "office",
"E": "hallway",
"F": "bathroom"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | office | Correct |
| Opus 5.5 | office | Correct |
| GLiDE v1.2 | office | Correct |
| Decider v1.1 | office | Correct |
| Jev | office | Correct |
| Microsoft Decision-1 | office | Correct |
| Clef | office | Correct |
| Luna Decisions | office | Correct |
| Clef Flash | office | Correct |
| Strands Decider | office | Correct |
| GLiNER2.5 Decide | garden | Incorrect |
| Laya | bedroom | Incorrect |
| CalDec v1 | bedroom | Incorrect |
Source and sharing
Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
"record": "2396",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "8aef832f2effd8a82ec4ad141f1661ef13cb36eff6299ee24879286e4d570d57"
}