RaindropA practical guide to choosing decision modelsGitHub ↗

Social cognition · question 8

Back to Social cognition · All capabilities

Question ID: bigtom/84e0b56db0941d7c5017
Source: BigToM
Input format: native choices

Explore this question’s altered versions

Context

Zara is a skilled tailor in a small Moroccan town. She is working on a custom-made dress for a client's wedding. Zara notices that the silk fabric she chose for the dress is of high quality, smooth and shiny. Zara believes that the silk fabric is indeed high quality, smooth, and shiny. While Zara is taking a break, her assistant accidentally spills a cup of coffee on the silk fabric, staining and damaging it. Zara does not notice the coffee spill on the silk fabric.

Question

Does Zara believe the silk fabric is high quality, smooth, and shiny, or stained and damaged?

Answer choices

  • A: Zara believes the silk fabric is stained and damaged.
  • B: Zara believes the silk fabric is high quality, smooth, and shiny.

Reference answer: Zara believes the silk fabric is high quality, smooth, and shiny. (B)

Exact evaluated input as JSON
{
  "state": "Zara is a skilled tailor in a small Moroccan town. She is working on a custom-made dress for a client's wedding. Zara notices that the silk fabric she chose for the dress is of high quality, smooth and shiny. Zara believes that the silk fabric is indeed high quality, smooth, and shiny. While Zara is taking a break, her assistant accidentally spills a cup of coffee on the silk fabric, staining and damaging it. Zara does not notice the coffee spill on the silk fabric.",
  "question": "Does Zara believe the silk fabric is high quality, smooth, and shiny, or stained and damaged?",
  "choices": {
    "A": "Zara believes the silk fabric is stained and damaged.",
    "B": "Zara believes the silk fabric is high quality, smooth, and shiny."
  }
}

Recorded model answers

Model Answer Outcome
Astra Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Opus 5.5 Zara believes the silk fabric is high quality, smooth, and shiny. Correct
GLiDE v1.2 Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Decider v1.1 Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Jev Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Microsoft Decision-1 Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Clef Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Luna Decisions Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Clef Flash Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Strands Decider Zara believes the silk fabric is high quality, smooth, and shiny. Correct
GLiNER2.5 Decide Zara believes the silk fabric is high quality, smooth, and shiny. Correct
Laya Zara believes the silk fabric is stained and damaged. Incorrect
CalDec v1 Zara believes the silk fabric is stained and damaged. Incorrect

Source and sharing

Attribution: Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
Terms: MIT. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/bigtom/bigtom.csv",
    "record": "182/forward-belief/false",
    "kind": "source_file_record"
  },
  "source_revision": "fe647d680bddb69f738519313bed625f9e93b549",
  "input_hash": "f7e74a5c4d022968455da12d47cb0618cdb1ead8b5b2cce718f632fd988df9d6"
}

Type to search questions, variants and documentation.