State tracking and updating · question 21
Back to State tracking and updating · All capabilities
Question ID: babi/2c8f31be4745073a29a6
Source: bAbI
Input format: adapted choices
Explore this question’s altered versions
Context
Daniel went to the office. John travelled to the kitchen. Mary journeyed to the bedroom. John picked up the apple there. John put down the apple. Sandra travelled to the office. Sandra went to the bathroom. Daniel grabbed the milk there. John journeyed to the office. John travelled to the bedroom. Daniel left the milk. Daniel grabbed the milk there. John moved to the garden. Daniel moved to the kitchen.
Question
Where is the milk?
Answer choices
- A: hallway
- B: bathroom
- C: kitchen
- D: office
- E: garden
- F: bedroom
Reference answer: kitchen (C)
Exact evaluated input as JSON
{
"state": "Daniel went to the office.\nJohn travelled to the kitchen.\nMary journeyed to the bedroom.\nJohn picked up the apple there.\nJohn put down the apple.\nSandra travelled to the office.\nSandra went to the bathroom.\nDaniel grabbed the milk there.\nJohn journeyed to the office.\nJohn travelled to the bedroom.\nDaniel left the milk.\nDaniel grabbed the milk there.\nJohn moved to the garden.\nDaniel moved to the kitchen.",
"question": "Where is the milk? ",
"choices": {
"A": "hallway",
"B": "bathroom",
"C": "kitchen",
"D": "office",
"E": "garden",
"F": "bedroom"
}
}
Recorded model answers
| Model | Answer | Outcome |
|---|---|---|
| Astra | kitchen | Correct |
| Opus 5.5 | kitchen | Correct |
| GLiDE v1.2 | kitchen | Correct |
| Decider v1.1 | kitchen | Correct |
| Jev | kitchen | Correct |
| Microsoft Decision-1 | kitchen | Correct |
| Clef | kitchen | Correct |
| Luna Decisions | kitchen | Correct |
| Clef Flash | kitchen | Correct |
| Strands Decider | kitchen | Correct |
| GLiNER2.5 Decide | kitchen | Correct |
| Laya | office | Incorrect |
| CalDec v1 | kitchen | Correct |
Source and sharing
Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.
Exact source locator and input fingerprint
{
"source_locator": {
"path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa2_two-supporting-facts_test.txt",
"record": "2979",
"kind": "source_file_record"
},
"source_revision": null,
"input_hash": "2366ffed9c2b67fb008e97263e87f78a624feea184900dac9ec0ff15022c5b89"
}