State tracking and updating · question 41
Back to State tracking and updating · All capabilities
Question ID: babi/94b56fad9ffb1fb6bffd
Source: bAbI
Input format: adapted choices
Explore this question’s altered versions
Context
Read the full context
Mary moved to the office. Sandra journeyed to the kitchen. Mary went back to the bedroom. Daniel moved to the office. Mary took the milk. Sandra went to the hallway. Mary picked up the football. Daniel went to the garden. Daniel moved to the bathroom. Sandra journeyed to the bedroom. Mary travelled to the hallway. John went to the bathroom. Mary discarded the milk there. Mary put down the football. Mary travelled to the kitchen. Sandra journeyed to the garden. Daniel went back to the hallway. Daniel grabbed the football. Sandra travelled to the hallway. Daniel left the football. Sandra went back to the garden. Mary went back to the bedroom. Sandra went back to the kitchen. Sandra travelled to the hallway. Mary went back to the office. Daniel travelled to the garden. Mary journeyed to the bathroom. Sandra grabbed the milk. Sandra discarded the milk. John moved to the hallway. Sandra went back to the office. John took the football. John dropped the football. Daniel journeyed to the kitchen. John travelled to the garden. Daniel went back to the bathroom. Sandra travelled to the bedroom. Daniel moved to the garden. Daniel travelled to the bathroom. Daniel went back to the office. Daniel went to the bathroom. Daniel went to the hallway. Daniel grabbed the football there. Mary journeyed to the garden. Sandra travelled to the garden. Mary went back to the bedroom. Mary travelled to the hallway. John journeyed to the hallway. Daniel grabbed the milk there. Sandra moved to the hallway. Sandra travelled to the garden. John grabbed the apple. John discarded the apple. John travelled to the office. John went back to the hallway. John got the apple. Sandra journeyed to the hallway. Mary went back to the office. Mary travelled to the kitchen. John dropped the apple. John took the apple. John moved to the kitchen. Daniel discarded the milk. Daniel moved to the garden. John left the apple there. Daniel travelled to the kitchen. Sandra travelled to the bathroom. John got the apple. Daniel left the football. John got the football. Sandra went back to the bedroom. John moved to the hallway. John discarded the apple. John discarded the football. Sandra went to the hallway. John travelled to the office. Sandra took the apple. Mary travelled to the hallway. Mary picked up the football. Mary moved to the bathroom. Daniel moved to the hallway. Mary went back to the garden. Sandra got the milk. John moved to the bedroom. Daniel travelled to the kitchen. Mary discarded the football. Mary moved to the hallway. John travelled to the hallway. Daniel went back to the bedroom. Sandra dropped the milk. Sandra got the milk. Mary moved to the bathroom. Sandra left the apple there. Sandra discarded the milk. Sandra travelled to the office. Sandra travelled to the kitchen. Daniel went back to the bathroom. Sandra moved to the hallway. John grabbed the milk. Daniel journeyed to the kitchen. Sandra travelled to the office. John took the apple. John travelled to the kitchen. John travelled to the office.
Question
Where was the apple before the office?
Answer choices
- A: kitchen
- B: bathroom
- C: bedroom
- D: hallway
- E: office
- F: garden
Reference answer: kitchen (A)
Exact evaluated input as JSON
{
"state": "Mary moved to the office.\nSandra journeyed to the kitchen.\nMary went back to the bedroom.\nDaniel moved to the office.\nMary took the milk.\nSandra went to the hallway.\nMary picked up the football.\nDaniel went to the garden.\nDaniel moved to the bathroom.\nSandra journeyed to the bedroom.\nMary travelled to the hallway.\nJohn went to the bathroom.\nMary discarded the milk there.\nMary put down the football.\nMary travelled to the kitchen.\nSandra journeyed to the garden.\nDaniel went back to the hallway.\nDaniel grabbed the football.\nSandra travelled to the hallway.\nDaniel left the football.\nSandra went back to the garden.\nMary went back to the bedroom.\nSandra went back to the kitchen.\nSandra travelled to the hallway.\nMary went back to the office.\nDaniel travelled to the garden.\nMary journeyed to the bathroom.\nSandra grabbed the milk.\nSandra discarded the milk.\nJohn moved to the hallway.\nSandra went back to the office.\nJohn took the football.\nJohn dropped the football.\nDaniel journeyed to the kitchen.\nJohn travelled to the garden.\nDaniel went back to the bathroom.\nSandra travelled to the bedroom.\nDaniel moved to the garden.\nDaniel travelled to the bathroom.\nDaniel went back to the office.\nDaniel went to the bathroom.\nDaniel went to the hallway.\nDaniel grabbed the football there.\nMary journeyed to the garden.\nSandra travelled to the garden.\nMary went back to the bedroom.\nMary travelled to the hallway.\nJohn journeyed to the hallway.\nDaniel grabbed the milk there.\nSandra moved to the hallway.\nSandra travelled to the garden.\nJohn grabbed the apple.\nJohn discarded the apple.\nJohn travelled to the office.\nJohn went back to the hallway.\nJohn got the apple.\nSandra journeyed to the hallway.\nMary went back to the office.\nMary travelled to the kitchen.\nJohn dropped the apple.\nJohn took the apple.\nJohn moved to the kitchen.\nDaniel discarded the milk.\nDaniel moved to the garden.\nJohn left the apple there.\nDaniel travelled to the kitchen.\nSandra travelled to the bathroom.\nJohn got the apple.\nDaniel left the football.\nJohn got the football.\nSandra went back to the bedroom.\nJohn moved to the hallway.\nJohn discarded the apple.\nJohn discarded the football.\nSandra went to the hallway.\nJohn travelled to the office.\nSandra took the apple.\nMary travelled to the hallway.\nMary picked up the football.\nMary moved to the bathroom.\nDaniel moved to the hallway.\nMary went back to the garden.\nSandra got the milk.\nJohn moved to the bedroom.\nDaniel travelled to the kitchen.\nMary discarded the football.\nMary moved to the hallway.\nJohn travelled to the hallway.\nDaniel went back to the bedroom.\nSandra dropped the milk.\nSandra got the milk.\nMary moved to the bathroom.\nSandra left the apple there.\nSandra discarded the milk.\nSandra travelled to the office.\nSandra travelled to the kitchen.\nDaniel went back to the bathroom.\nSandra moved to the hallway.\nJohn grabbed the milk.\nDaniel journeyed to the kitchen.\nSandra travelled to the office.\nJohn took the apple.\nJohn travelled to the kitchen.\nJohn travelled to the office.",
"question": "Where was the apple before the office? ",
"choices": {
"A": "kitchen",
"B": "bathroom",
"C": "bedroom",
"D": "hallway",
"E": "office",
"F": "garden"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | kitchen | Correct |
| Opus 5.5 | kitchen | Correct |
| GLiDE v1.2 | kitchen | Correct |
| Decider v1.1 | kitchen | Correct |
| Jev | kitchen | Correct |
| Microsoft Decision-1 | kitchen | Correct |
| Clef | kitchen | Correct |
| Luna Decisions | hallway | Incorrect |
| Clef Flash | kitchen | Correct |
| Strands Decider | kitchen | Correct |
| GLiNER2.5 Decide | office | Incorrect |
| Laya | kitchen | Correct |
| CalDec v1 | office | Incorrect |
Source and sharing
Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa3_three-supporting-facts_test.txt",
"record": "9189",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "71751bd6edba71de1355abb2f866eb6fedeea21c9405c7080f533393b2893690"
}