RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 21

Back to State tracking and updating · All capabilities

Question ID: babi/2c8f31be4745073a29a6
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Daniel went to the office. John travelled to the kitchen. Mary journeyed to the bedroom. John picked up the apple there. John put down the apple. Sandra travelled to the office. Sandra went to the bathroom. Daniel grabbed the milk there. John journeyed to the office. John travelled to the bedroom. Daniel left the milk. Daniel grabbed the milk there. John moved to the garden. Daniel moved to the kitchen.

Question

Where is the milk?

Answer choices

  • A: hallway
  • B: bathroom
  • C: kitchen
  • D: office
  • E: garden
  • F: bedroom

Reference answer: kitchen (C)

Exact evaluated input as JSON
{
  "state": "Daniel went to the office.\nJohn travelled to the kitchen.\nMary journeyed to the bedroom.\nJohn picked up the apple there.\nJohn put down the apple.\nSandra travelled to the office.\nSandra went to the bathroom.\nDaniel grabbed the milk there.\nJohn journeyed to the office.\nJohn travelled to the bedroom.\nDaniel left the milk.\nDaniel grabbed the milk there.\nJohn moved to the garden.\nDaniel moved to the kitchen.",
  "question": "Where is the milk? ",
  "choices": {
    "A": "hallway",
    "B": "bathroom",
    "C": "kitchen",
    "D": "office",
    "E": "garden",
    "F": "bedroom"
  }
}

Recorded model answers

Model Answer Outcome
Astra kitchen Correct
Opus 5.5 kitchen Correct
GLiDE v1.2 kitchen Correct
Decider v1.1 kitchen Correct
Jev kitchen Correct
Microsoft Decision-1 kitchen Correct
Clef kitchen Correct
Luna Decisions kitchen Correct
Clef Flash kitchen Correct
Strands Decider kitchen Correct
GLiNER2.5 Decide kitchen Correct
Laya office Incorrect
CalDec v1 kitchen Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa2_two-supporting-facts_test.txt",
    "record": "2979",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "2366ffed9c2b67fb008e97263e87f78a624feea184900dac9ec0ff15022c5b89"
}

Type to search questions, variants and documentation.