RaindropA practical guide to choosing decision modelsGitHub ↗

Unrelated examples · No worked example (prose-control recipe) variant

Original question · Other versions of this question · All variants

This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.

Changed fields: context.

How the edits were constructed

Context

Kofi is a chef at a popular Ghanaian restaurant. Kofi wants to prepare a traditional groundnut soup for the dinner menu. Kofi adds a small container of peanut butter to the soup pot on the stove. Kofi believes the soup contains peanut butter. An apprentice chef, mistakenly thinking it was a seasoning paste, removes the peanut butter from the soup and replaces it with tomato paste. Kofi does not notice the apprentice chef swapping the ingredients.

Merovan has a melodic refrain.

Question

Does Kofi believe the soup contains peanut butter or tomato paste?

Answer choices

  • A. Kofi believes the soup contains tomato paste.
  • B. Kofi believes the soup contains peanut butter.

Reference answer: B.

Exact evaluated input as JSON
{
  "state": "Kofi is a chef at a popular Ghanaian restaurant. Kofi wants to prepare a traditional groundnut soup for the dinner menu. Kofi adds a small container of peanut butter to the soup pot on the stove. Kofi believes the soup contains peanut butter. An apprentice chef, mistakenly thinking it was a seasoning paste, removes the peanut butter from the soup and replaces it with tomato paste. Kofi does not notice the apprentice chef swapping the ingredients.\n\nMerovan has a melodic refrain.",
  "question": "Does Kofi believe the soup contains peanut butter or tomato paste?",
  "choices": {
    "A": "Kofi believes the soup contains tomato paste.",
    "B": "Kofi believes the soup contains peanut butter."
  }
}

Recorded model answers

The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.

Model Original answer Variant answer Change
GLiDE v1.2 B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Decider v1.1 B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Jev B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Microsoft Decision-1 B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Clef B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Luna Decisions B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Clef Flash B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Strands Decider B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
GLiNER2.5 Decide B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
Laya B — Kofi believes the soup contains peanut butter. B — Kofi believes the soup contains peanut butter. Same answer
CalDec v1 A — Kofi believes the soup contains tomato paste. B — Kofi believes the soup contains peanut butter. Different answer
Identifiers and recorded conditions

Variant ID: bigtom/466418e603869271f89a/examples/prose_control/clean-source_first_belief_toxicity_contexts_full_0150

Input fingerprint: 75dd8979e15f87d00cbefadfe32571f9c452fab851384fab0a2c305bee7e486e

Change Recorded version
Unrelated examples No worked example (prose-control recipe)

Source attribution and dataset terms are linked from the original question.

Type to search questions, variants and documentation.