RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating · question 2

Back to State tracking and updating · All capabilities

Question ID: babi/25a8cecfd568bcdf37f2
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Daniel went to the hallway. Daniel went to the bedroom. John journeyed to the office. Mary went back to the bedroom. Mary travelled to the office. John travelled to the bedroom. Mary went to the hallway. Mary went back to the kitchen. Daniel moved to the hallway. Mary moved to the office.

Question

Where is Mary?

Answer choices

  • A: office
  • B: bathroom
  • C: garden
  • D: kitchen
  • E: bedroom
  • F: hallway

Reference answer: office (A)

Exact evaluated input as JSON
{
  "state": "Daniel went to the hallway.\nDaniel went to the bedroom.\nJohn journeyed to the office.\nMary went back to the bedroom.\nMary travelled to the office.\nJohn travelled to the bedroom.\nMary went to the hallway.\nMary went back to the kitchen.\nDaniel moved to the hallway.\nMary moved to the office.",
  "question": "Where is Mary? ",
  "choices": {
    "A": "office",
    "B": "bathroom",
    "C": "garden",
    "D": "kitchen",
    "E": "bedroom",
    "F": "hallway"
  }
}

Recorded model answers

Model Answer Outcome
Astra office Correct
Opus 5.5 office Correct
GLiDE v1.2 office Correct
Decider v1.1 office Correct
Jev office Correct
Microsoft Decision-1 office Correct
Clef office Correct
Luna Decisions office Correct
Clef Flash office Correct
Strands Decider office Correct
GLiNER2.5 Decide office Correct
Laya office Correct
CalDec v1 office Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
    "record": "2579",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "ad37abcb0f1e49117d35c30b1f3c5842d1f5c1ae54bbf78aec182a96d986f137"
}

Type to search questions, variants and documentation.