RaindropA practical guide to choosing decision modelsGitHub ↗

Invented names · Use invented names variant

Original question · Other versions of this question · All variants

This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.

Changed fields: context.

How the edits were constructed

Context

Tavren journeyed to the bedroom. Tavren went back to the hallway. Nelora travelled to the hallway. Vaelis went to the hallway. Vaelis travelled to the garden. Vaelis grabbed the milk there. Tavren moved to the office. Nelora moved to the kitchen. Vaelis put down the milk there. Nelora went back to the bedroom. Vaelis grabbed the milk there. Tavren journeyed to the hallway. Vaelis travelled to the bathroom. Nelora journeyed to the hallway. Zorvik travelled to the bathroom. Zorvik journeyed to the hallway. Vaelis took the apple. Vaelis moved to the office. Tavren travelled to the office. Vaelis moved to the hallway. Nelora went to the bedroom. Tavren went back to the bathroom. Vaelis moved to the garden. Vaelis travelled to the bathroom. Tavren travelled to the kitchen. Vaelis discarded the milk. Zorvik went to the bathroom. Tavren went to the bathroom.

Question

Where was the milk before the garden?

Answer choices

  • A. office
  • B. hallway
  • C. garden
  • D. bathroom
  • E. bedroom
  • F. kitchen

Reference answer: B.

Exact evaluated input as JSON
{
  "state": "Tavren journeyed to the bedroom.\nTavren went back to the hallway.\nNelora travelled to the hallway.\nVaelis went to the hallway.\nVaelis travelled to the garden.\nVaelis grabbed the milk there.\nTavren moved to the office.\nNelora moved to the kitchen.\nVaelis put down the milk there.\nNelora went back to the bedroom.\nVaelis grabbed the milk there.\nTavren journeyed to the hallway.\nVaelis travelled to the bathroom.\nNelora journeyed to the hallway.\nZorvik travelled to the bathroom.\nZorvik journeyed to the hallway.\nVaelis took the apple.\nVaelis moved to the office.\nTavren travelled to the office.\nVaelis moved to the hallway.\nNelora went to the bedroom.\nTavren went back to the bathroom.\nVaelis moved to the garden.\nVaelis travelled to the bathroom.\nTavren travelled to the kitchen.\nVaelis discarded the milk.\nZorvik went to the bathroom.\nTavren went to the bathroom.",
  "question": "Where was the milk before the garden? ",
  "choices": {
    "A": "office",
    "B": "hallway",
    "C": "garden",
    "D": "bathroom",
    "E": "bedroom",
    "F": "kitchen"
  }
}

Recorded model answers

The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.

Model Original answer Variant answer Change
GLiDE v1.2 B — hallway B — hallway Same answer
Decider v1.1 B — hallway B — hallway Same answer
Jev B — hallway C — garden Different answer
Microsoft Decision-1 D — bathroom D — bathroom Same answer
Clef D — bathroom D — bathroom Same answer
Luna Decisions B — hallway B — hallway Same answer
Clef Flash C — garden C — garden Same answer
Strands Decider D — bathroom C — garden Different answer
GLiNER2.5 Decide D — bathroom D — bathroom Same answer
Laya A — office A — office Same answer
CalDec v1 A — office B — hallway Different answer
Identifiers and recorded conditions

Variant ID: babi/ca4805a20f60de636d46/unfamiliar_names/invented

Input fingerprint: 8ad33a840fb2bdf55995597285f19df4d3557b5ffc33ca03e7fbb11d0c1e7388

Change Recorded version
Invented names Use invented names

Source attribution and dataset terms are linked from the original question.

Type to search questions, variants and documentation.