RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 55

Back to State tracking and updating · All capabilities

Question ID: babi/59637e1b402bb9904895
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Daniel travelled to the office. Sandra went back to the office. Mary went to the office. Daniel moved to the bathroom. John went to the hallway. Sandra moved to the bedroom. Sandra travelled to the office. John went back to the garden.

Question

Where is Sandra?

Answer choices

  • A: garden
  • B: bedroom
  • C: kitchen
  • D: office
  • E: hallway
  • F: bathroom

Reference answer: office (D)

Exact evaluated input as JSON
{
  "state": "Daniel travelled to the office.\nSandra went back to the office.\nMary went to the office.\nDaniel moved to the bathroom.\nJohn went to the hallway.\nSandra moved to the bedroom.\nSandra travelled to the office.\nJohn went back to the garden.",
  "question": "Where is Sandra? ",
  "choices": {
    "A": "garden",
    "B": "bedroom",
    "C": "kitchen",
    "D": "office",
    "E": "hallway",
    "F": "bathroom"
  }
}

Recorded model answers

Model Answer Outcome
Astra office Correct
Opus 5.5 office Correct
GLiDE v1.2 office Correct
Decider v1.1 office Correct
Jev office Correct
Microsoft Decision-1 office Correct
Clef office Correct
Luna Decisions office Correct
Clef Flash office Correct
Strands Decider office Correct
GLiNER2.5 Decide garden Incorrect
Laya bedroom Incorrect
CalDec v1 bedroom Incorrect

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
    "record": "2396",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "8aef832f2effd8a82ec4ad141f1661ef13cb36eff6299ee24879286e4d570d57"
}

Type to search questions, variants and documentation.