RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 15

Back to Retrieval · All capabilities

Question ID: babi/362577e03d1a583615b5
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Mary journeyed to the kitchen. Sandra went back to the garden.

Question

Where is Sandra?

Answer choices

  • A: bedroom
  • B: kitchen
  • C: hallway
  • D: office
  • E: bathroom
  • F: garden

Reference answer: garden (F)

Exact evaluated input as JSON
{
  "state": "Mary journeyed to the kitchen.\nSandra went back to the garden.",
  "question": "Where is Sandra? ",
  "choices": {
    "A": "bedroom",
    "B": "kitchen",
    "C": "hallway",
    "D": "office",
    "E": "bathroom",
    "F": "garden"
  }
}

Recorded model answers

Model Answer Outcome
Astra garden Correct
Opus 5.5 garden Correct
GLiDE v1.2 garden Correct
Decider v1.1 garden Correct
Jev garden Correct
Microsoft Decision-1 garden Correct
Clef garden Correct
Luna Decisions garden Correct
Clef Flash garden Correct
Strands Decider garden Correct
GLiNER2.5 Decide garden Correct
Laya garden Correct
CalDec v1 garden Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
    "record": "1397",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "3a6e606554ae27e357ce43bbbc3a2ae4f1d8d0bce84d85e08f191db56c62add3"
}

Type to search questions, variants and documentation.