RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 64

Back to Relations · All capabilities

Question ID: clutrr/522aec94c070e77f1132
Source: CLUTRR
Input format: native choices

Explore this question’s altered versions

Context

[Patricia] took her son [Timothy] to school because he missed the bus. [Patricia] dropped her son, [Timothy], off for soccer practice. Afterwards she went home to make a cake for her daughter, [Christina]. [Christina] took her sister, [Connie], out to dinner for her birthday.

Question

What is Patricia’s relationship to Connie?

Answer choices

  • aunt: aunt
  • brother: brother
  • daughter: daughter
  • daughter-in-law: daughter-in-law
  • father: father
  • father-in-law: father-in-law
  • granddaughter: granddaughter
  • grandfather: grandfather
  • grandmother: grandmother
  • grandson: grandson
  • mother: mother
  • mother-in-law: mother-in-law
  • nephew: nephew
  • niece: niece
  • sister: sister
  • son: son
  • son-in-law: son-in-law
  • uncle: uncle

Reference answer: mother (mother)

Exact evaluated input as JSON
{
  "state": "[Patricia] took her son [Timothy] to school because he missed the bus. [Patricia] dropped her son, [Timothy], off for soccer practice. Afterwards she went home to make a cake for her daughter, [Christina]. [Christina] took her sister, [Connie], out to dinner for her birthday.",
  "question": "What is Patricia’s relationship to Connie?",
  "choices": {
    "aunt": "aunt",
    "brother": "brother",
    "daughter": "daughter",
    "daughter-in-law": "daughter-in-law",
    "father": "father",
    "father-in-law": "father-in-law",
    "granddaughter": "granddaughter",
    "grandfather": "grandfather",
    "grandmother": "grandmother",
    "grandson": "grandson",
    "mother": "mother",
    "mother-in-law": "mother-in-law",
    "nephew": "nephew",
    "niece": "niece",
    "sister": "sister",
    "son": "son",
    "son-in-law": "son-in-law",
    "uncle": "uncle"
  }
}

Recorded model answers

Model Answer Outcome
Astra mother Correct
Opus 5.5 mother Correct
GLiDE v1.2 mother Correct
Decider v1.1 mother Correct
Jev mother Correct
Microsoft Decision-1 aunt Incorrect
Clef aunt Incorrect
Luna Decisions mother Correct
Clef Flash aunt Incorrect
Strands Decider mother Correct
GLiNER2.5 Decide sister Incorrect
Laya sister Incorrect
CalDec v1 sister Incorrect

Source and sharing

Attribution: Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "gen_train23_test2to10/test.csv",
    "record": "214",
    "kind": "source_file_record"
  },
  "source_revision": "e5b496941e91abb7c319d2618a3ce96752bc4ab7",
  "input_hash": "9e2346d76baedec4a6edf5b6766c161c7a24b8ab72e941c4dc24b7b0c2700669"
}

Type to search questions, variants and documentation.