RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 28

Back to Retrieval · All capabilities

Question ID: babi/40d6874c302e41b99cdb
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Mary journeyed to the hallway. John went back to the bathroom.

Question

Where is Mary?

Answer choices

  • A: garden
  • B: kitchen
  • C: bathroom
  • D: hallway
  • E: office
  • F: bedroom

Reference answer: hallway (D)

Exact evaluated input as JSON
{
  "state": "Mary journeyed to the hallway.\nJohn went back to the bathroom.",
  "question": "Where is Mary? ",
  "choices": {
    "A": "garden",
    "B": "kitchen",
    "C": "bathroom",
    "D": "hallway",
    "E": "office",
    "F": "bedroom"
  }
}

Recorded model answers

Model Answer Outcome
Astra hallway Correct
Opus 5.5 hallway Correct
GLiDE v1.2 hallway Correct
Decider v1.1 hallway Correct
Jev hallway Correct
Microsoft Decision-1 hallway Correct
Clef hallway Correct
Luna Decisions hallway Correct
Clef Flash hallway Correct
Strands Decider hallway Correct
GLiNER2.5 Decide hallway Correct
Laya hallway Correct
CalDec v1 hallway Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_train.txt",
    "record": "2522",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "9191748ba55cb4681914453ed583faf6cebf6d76fedc84b0315477d785da734c"
}

Type to search questions, variants and documentation.