RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 40

Back to Retrieval · All capabilities

Question ID: babi/cc3432e648d116c87be2
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Sandra moved to the bedroom. Daniel journeyed to the garden. Mary moved to the bedroom. Sandra went back to the hallway. John went to the office. Sandra went back to the bathroom.

Question

Where is Daniel?

Answer choices

  • A: bathroom
  • B: garden
  • C: kitchen
  • D: office
  • E: bedroom
  • F: hallway

Reference answer: garden (B)

Exact evaluated input as JSON
{
  "state": "Sandra moved to the bedroom.\nDaniel journeyed to the garden.\nMary moved to the bedroom.\nSandra went back to the hallway.\nJohn went to the office.\nSandra went back to the bathroom.",
  "question": "Where is Daniel? ",
  "choices": {
    "A": "bathroom",
    "B": "garden",
    "C": "kitchen",
    "D": "office",
    "E": "bedroom",
    "F": "hallway"
  }
}

Recorded model answers

Model Answer Outcome
Astra garden Correct
Opus 5.5 garden Correct
GLiDE v1.2 garden Correct
Decider v1.1 garden Correct
Jev garden Correct
Microsoft Decision-1 garden Correct
Clef garden Correct
Luna Decisions garden Correct
Clef Flash garden Correct
Strands Decider garden Correct
GLiNER2.5 Decide bathroom Incorrect
Laya bathroom Incorrect
CalDec v1 bathroom Incorrect

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_test.txt",
    "record": "2168",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "28cfacf0ff8af2f36ed71c3a365e599a59af4a2445f0bec8968f49a90b27438c"
}

Type to search questions, variants and documentation.