RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 61

Back to State tracking and updating · All capabilities

Question ID: babi/64437a914bee36f9a192
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Sandra moved to the kitchen. John travelled to the office. Sandra went back to the garden. John went to the garden. Daniel travelled to the garden. Sandra moved to the hallway. Daniel went to the kitchen. Daniel journeyed to the office.

Question

Where is John?

Answer choices

  • A: bathroom
  • B: office
  • C: hallway
  • D: kitchen
  • E: bedroom
  • F: garden

Reference answer: garden (F)

Exact evaluated input as JSON
{
  "state": "Sandra moved to the kitchen.\nJohn travelled to the office.\nSandra went back to the garden.\nJohn went to the garden.\nDaniel travelled to the garden.\nSandra moved to the hallway.\nDaniel went to the kitchen.\nDaniel journeyed to the office.",
  "question": "Where is John? ",
  "choices": {
    "A": "bathroom",
    "B": "office",
    "C": "hallway",
    "D": "kitchen",
    "E": "bedroom",
    "F": "garden"
  }
}

Recorded model answers

Model Answer Outcome
Astra garden Correct
Opus 5.5 garden Correct
GLiDE v1.2 garden Correct
Decider v1.1 garden Correct
Jev garden Correct
Microsoft Decision-1 garden Correct
Clef garden Correct
Luna Decisions garden Correct
Clef Flash garden Correct
Strands Decider garden Correct
GLiNER2.5 Decide garden Correct
Laya office Incorrect
CalDec v1 office Incorrect

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
    "record": "2696",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "142bbc9c3f803f9a5e5290c9897be7c5388a887ba8da2f1eb1de46dca966a539"
}

Type to search questions, variants and documentation.