RaindropA practical guide to choosing decision modelsGitHub ↗

Retrieval · question 32

Back to Retrieval · All capabilities

Question ID: babi/73c788d453e9fe98e2b0
Source: bAbI
Input format: adapted choices

Explore this question’s altered versions

Context

Mary journeyed to the garden. Sandra went back to the office.

Question

Where is Sandra?

Answer choices

  • A: kitchen
  • B: office
  • C: bathroom
  • D: hallway
  • E: garden
  • F: bedroom

Reference answer: office (B)

Exact evaluated input as JSON
{
  "state": "Mary journeyed to the garden.\nSandra went back to the office.",
  "question": "Where is Sandra? ",
  "choices": {
    "A": "kitchen",
    "B": "office",
    "C": "bathroom",
    "D": "hallway",
    "E": "garden",
    "F": "bedroom"
  }
}

Recorded model answers

Model Answer Outcome
Astra office Correct
Opus 5.5 office Correct
GLiDE v1.2 office Correct
Decider v1.1 office Correct
Jev office Correct
Microsoft Decision-1 office Correct
Clef office Correct
Luna Decisions office Correct
Clef Flash office Correct
Strands Decider office Correct
GLiNER2.5 Decide office Correct
Laya office Correct
CalDec v1 office Correct

Source and sharing

Attribution: Facebook, Inc.; bAbI authors.
Terms: CC-BY-3.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "extracted/tasks.tar.gz/tasks_1-20_v1-2/en/qa1_single-supporting-fact_train.txt",
    "record": "242",
    "kind": "source_file_record"
  },
  "source_revision": null,
  "input_hash": "0c2188a7f4522c3b1835285620e614989bb35747811392cabeb0cf4ea03a1265"
}

Type to search questions, variants and documentation.