RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 36

Back to Retrieval · All capabilities

Question ID: babi/9402ed5547bab520026c
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

John travelled to the bathroom. Daniel went to the hallway. John moved to the kitchen. Mary journeyed to the kitchen. Daniel travelled to the bathroom. John travelled to the bathroom. Daniel went to the garden. Sandra travelled to the hallway.

Question

Where is Sandra?

Answer choices

  • A: office
  • B: garden
  • C: hallway
  • D: bathroom
  • E: kitchen
  • F: bedroom

Reference answer: hallway (C)

Exact evaluated input as JSON
{
  "state": "John travelled to the bathroom.\nDaniel went to the hallway.\nJohn moved to the kitchen.\nMary journeyed to the kitchen.\nDaniel travelled to the bathroom.\nJohn travelled to the bathroom.\nDaniel went to the garden.\nSandra travelled to the hallway.",
  "question": "Where is Sandra? ",
  "choices": {
    "A": "office",
    "B": "garden",
    "C": "hallway",
    "D": "bathroom",
    "E": "kitchen",
    "F": "bedroom"
  }
}

Recorded model answers

Model Answer Outcome
Astra hallway Correct
Opus 5.5 hallway Correct
GLiDE v1.2 hallway Correct
Decider v1.1 hallway Correct
Jev hallway Correct
Microsoft Decision-1 hallway Correct
Clef hallway Correct
Luna Decisions hallway Correct
Clef Flash hallway Correct
Strands Decider hallway Correct
GLiNER2.5 Decide hallway Correct
Laya hallway Correct
CalDec v1 hallway Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_train.txt",
    "record": "2561",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "4ad0cad0b3d2c5dafa1551dd261129a411330fac1cb2dc6eb426bc12f491a87d"
}

Type to search questions, variants and documentation.