State tracking and updating · question 45
Back to State tracking and updating · All capabilities
Question ID: babi/0ebef6723a11f5e9f14c
Source: bAbI
Input format: adapted choices
Explore this question’s altered versions
Context
John moved to the garden. John went to the bedroom. Mary went back to the kitchen. Sandra moved to the garden. Daniel journeyed to the hallway. John journeyed to the kitchen. John journeyed to the office. Daniel travelled to the garden. John journeyed to the garden. Sandra took the milk. Sandra put down the milk. Mary moved to the bathroom. Daniel grabbed the milk there. John moved to the bathroom. Daniel put down the milk. John got the football. Mary travelled to the office. Sandra got the milk. Sandra left the milk. Sandra journeyed to the kitchen. John left the football there. John went to the kitchen. Daniel went to the kitchen. John moved to the hallway. Daniel went to the hallway. Mary journeyed to the bathroom. Mary took the football. Mary moved to the office. Daniel took the apple. Mary discarded the football there. Sandra travelled to the bedroom. Daniel put down the apple. Daniel took the apple. Daniel moved to the garden. Daniel moved to the hallway. Mary grabbed the football. Mary travelled to the bedroom. Daniel went to the garden. Sandra journeyed to the bathroom. Mary discarded the football there. John moved to the kitchen. Daniel discarded the apple.
Question
Where was the apple before the hallway?
Answer choices
- A: bathroom
- B: bedroom
- C: office
- D: kitchen
- E: garden
- F: hallway
Reference answer: garden (E)
Exact evaluated input as JSON
{
"state": "John moved to the garden.\nJohn went to the bedroom.\nMary went back to the kitchen.\nSandra moved to the garden.\nDaniel journeyed to the hallway.\nJohn journeyed to the kitchen.\nJohn journeyed to the office.\nDaniel travelled to the garden.\nJohn journeyed to the garden.\nSandra took the milk.\nSandra put down the milk.\nMary moved to the bathroom.\nDaniel grabbed the milk there.\nJohn moved to the bathroom.\nDaniel put down the milk.\nJohn got the football.\nMary travelled to the office.\nSandra got the milk.\nSandra left the milk.\nSandra journeyed to the kitchen.\nJohn left the football there.\nJohn went to the kitchen.\nDaniel went to the kitchen.\nJohn moved to the hallway.\nDaniel went to the hallway.\nMary journeyed to the bathroom.\nMary took the football.\nMary moved to the office.\nDaniel took the apple.\nMary discarded the football there.\nSandra travelled to the bedroom.\nDaniel put down the apple.\nDaniel took the apple.\nDaniel moved to the garden.\nDaniel moved to the hallway.\nMary grabbed the football.\nMary travelled to the bedroom.\nDaniel went to the garden.\nSandra journeyed to the bathroom.\nMary discarded the football there.\nJohn moved to the kitchen.\nDaniel discarded the apple.",
"question": "Where was the apple before the hallway? ",
"choices": {
"A": "bathroom",
"B": "bedroom",
"C": "office",
"D": "kitchen",
"E": "garden",
"F": "hallway"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | garden | Correct |
| Opus 5.5 | garden | Correct |
| GLiDE v1.2 | garden | Correct |
| Decider v1.1 | garden | Correct |
| Jev | garden | Correct |
| Microsoft Decision-1 | garden | Correct |
| Clef | garden | Correct |
| Luna Decisions | garden | Correct |
| Clef Flash | hallway | Incorrect |
| Strands Decider | garden | Correct |
| GLiNER2.5 Decide | garden | Correct |
| Laya | bathroom | Incorrect |
| CalDec v1 | office | Incorrect |
Source and sharing
Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa3_three-supporting-facts_train.txt",
"record": "4049",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "6a06d6fa36f44264979b8c4ac252e0702480342412d678329f088812d17f46be"
}