RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 45

Back to State tracking and updating · All capabilities

Question ID: babi/0ebef6723a11f5e9f14c
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

John moved to the garden. John went to the bedroom. Mary went back to the kitchen. Sandra moved to the garden. Daniel journeyed to the hallway. John journeyed to the kitchen. John journeyed to the office. Daniel travelled to the garden. John journeyed to the garden. Sandra took the milk. Sandra put down the milk. Mary moved to the bathroom. Daniel grabbed the milk there. John moved to the bathroom. Daniel put down the milk. John got the football. Mary travelled to the office. Sandra got the milk. Sandra left the milk. Sandra journeyed to the kitchen. John left the football there. John went to the kitchen. Daniel went to the kitchen. John moved to the hallway. Daniel went to the hallway. Mary journeyed to the bathroom. Mary took the football. Mary moved to the office. Daniel took the apple. Mary discarded the football there. Sandra travelled to the bedroom. Daniel put down the apple. Daniel took the apple. Daniel moved to the garden. Daniel moved to the hallway. Mary grabbed the football. Mary travelled to the bedroom. Daniel went to the garden. Sandra journeyed to the bathroom. Mary discarded the football there. John moved to the kitchen. Daniel discarded the apple.

Question

Where was the apple before the hallway?

Answer choices

  • A: bathroom
  • B: bedroom
  • C: office
  • D: kitchen
  • E: garden
  • F: hallway

Reference answer: garden (E)

Exact evaluated input as JSON
{
  "state": "John moved to the garden.\nJohn went to the bedroom.\nMary went back to the kitchen.\nSandra moved to the garden.\nDaniel journeyed to the hallway.\nJohn journeyed to the kitchen.\nJohn journeyed to the office.\nDaniel travelled to the garden.\nJohn journeyed to the garden.\nSandra took the milk.\nSandra put down the milk.\nMary moved to the bathroom.\nDaniel grabbed the milk there.\nJohn moved to the bathroom.\nDaniel put down the milk.\nJohn got the football.\nMary travelled to the office.\nSandra got the milk.\nSandra left the milk.\nSandra journeyed to the kitchen.\nJohn left the football there.\nJohn went to the kitchen.\nDaniel went to the kitchen.\nJohn moved to the hallway.\nDaniel went to the hallway.\nMary journeyed to the bathroom.\nMary took the football.\nMary moved to the office.\nDaniel took the apple.\nMary discarded the football there.\nSandra travelled to the bedroom.\nDaniel put down the apple.\nDaniel took the apple.\nDaniel moved to the garden.\nDaniel moved to the hallway.\nMary grabbed the football.\nMary travelled to the bedroom.\nDaniel went to the garden.\nSandra journeyed to the bathroom.\nMary discarded the football there.\nJohn moved to the kitchen.\nDaniel discarded the apple.",
  "question": "Where was the apple before the hallway? ",
  "choices": {
    "A": "bathroom",
    "B": "bedroom",
    "C": "office",
    "D": "kitchen",
    "E": "garden",
    "F": "hallway"
  }
}

Recorded model answers

Model Answer Outcome
Astra garden Correct
Opus 5.5 garden Correct
GLiDE v1.2 garden Correct
Decider v1.1 garden Correct
Jev garden Correct
Microsoft Decision-1 garden Correct
Clef garden Correct
Luna Decisions garden Correct
Clef Flash hallway Incorrect
Strands Decider garden Correct
GLiNER2.5 Decide garden Correct
Laya bathroom Incorrect
CalDec v1 office Incorrect

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa3_three-supporting-facts_train.txt",
    "record": "4049",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "6a06d6fa36f44264979b8c4ac252e0702480342412d678329f088812d17f46be"
}

Type to search questions, variants and documentation.