RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 18

Back to State tracking and updating · All capabilities

Question ID: babi/93021258c6ea2e72208f
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Mary got the milk. John journeyed to the office. Mary discarded the milk. Mary took the milk. John went to the bathroom. Mary left the milk. Daniel went to the garden. Sandra moved to the garden. Sandra went to the bathroom. Daniel went back to the bedroom. Sandra got the football. Sandra discarded the football. Mary got the milk there. John moved to the office. Daniel went to the hallway. Sandra journeyed to the garden. Mary left the milk. Daniel went back to the bedroom. Daniel took the milk. Daniel left the milk. Mary moved to the kitchen. Mary went back to the bathroom. John journeyed to the bedroom. Daniel went back to the office. Mary took the football. Mary discarded the football. Mary grabbed the football. John journeyed to the office. Mary put down the football. Mary took the football. Mary journeyed to the hallway. Mary left the football. Mary picked up the apple. Sandra moved to the hallway. Sandra took the football. Mary went back to the garden. Mary left the apple. Sandra went to the office. Daniel journeyed to the garden. John went back to the bathroom. Mary went back to the office. Daniel grabbed the apple. Mary went back to the bathroom. John travelled to the kitchen. Mary journeyed to the kitchen. Mary moved to the bathroom. Sandra went to the kitchen. Sandra left the football.

Question

Where was the football before the kitchen?

Answer choices

  • A: bathroom
  • B: hallway
  • C: kitchen
  • D: bedroom
  • E: office
  • F: garden

Reference answer: office (E)

Exact evaluated input as JSON
{
  "state": "Mary got the milk.\nJohn journeyed to the office.\nMary discarded the milk.\nMary took the milk.\nJohn went to the bathroom.\nMary left the milk.\nDaniel went to the garden.\nSandra moved to the garden.\nSandra went to the bathroom.\nDaniel went back to the bedroom.\nSandra got the football.\nSandra discarded the football.\nMary got the milk there.\nJohn moved to the office.\nDaniel went to the hallway.\nSandra journeyed to the garden.\nMary left the milk.\nDaniel went back to the bedroom.\nDaniel took the milk.\nDaniel left the milk.\nMary moved to the kitchen.\nMary went back to the bathroom.\nJohn journeyed to the bedroom.\nDaniel went back to the office.\nMary took the football.\nMary discarded the football.\nMary grabbed the football.\nJohn journeyed to the office.\nMary put down the football.\nMary took the football.\nMary journeyed to the hallway.\nMary left the football.\nMary picked up the apple.\nSandra moved to the hallway.\nSandra took the football.\nMary went back to the garden.\nMary left the apple.\nSandra went to the office.\nDaniel journeyed to the garden.\nJohn went back to the bathroom.\nMary went back to the office.\nDaniel grabbed the apple.\nMary went back to the bathroom.\nJohn travelled to the kitchen.\nMary journeyed to the kitchen.\nMary moved to the bathroom.\nSandra went to the kitchen.\nSandra left the football.",
  "question": "Where was the football before the kitchen? ",
  "choices": {
    "A": "bathroom",
    "B": "hallway",
    "C": "kitchen",
    "D": "bedroom",
    "E": "office",
    "F": "garden"
  }
}

Recorded model answers

Model Answer Outcome
Astra office Correct
Opus 5.5 office Correct
GLiDE v1.2 office Correct
Decider v1.1 office Correct
Jev office Correct
Microsoft Decision-1 office Correct
Clef office Correct
Luna Decisions hallway Incorrect
Clef Flash hallway Incorrect
Strands Decider hallway Incorrect
GLiNER2.5 Decide garden Incorrect
Laya bathroom Incorrect
CalDec v1 bathroom Incorrect

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa3_three-supporting-facts_test.txt",
    "record": "15258",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "9fb712f228f398545a9c3728b715f2c867cba3ce8ee16c495cb8cf82c7f2267c"
}

Type to search questions, variants and documentation.