RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 32

Back to Relations · All capabilities

Question ID: clutrr/865f24d475598cdce57e
Source: CLUTRR
Input format: native choices

Explore this question’s altered versions

Context

[Matthew] likes to watch baseball with his brother. His name is [Anthony]. [Stanley] had a conversation with his grandson, [Matthew]. [Stanley] asked his wife [Molly] to keep her distance.

Question

What is Anthony’s relationship to Molly?

Answer choices

  • aunt: aunt
  • brother: brother
  • daughter: daughter
  • daughter-in-law: daughter-in-law
  • father: father
  • father-in-law: father-in-law
  • granddaughter: granddaughter
  • grandfather: grandfather
  • grandmother: grandmother
  • grandson: grandson
  • mother: mother
  • mother-in-law: mother-in-law
  • nephew: nephew
  • niece: niece
  • sister: sister
  • son: son
  • son-in-law: son-in-law
  • uncle: uncle

Reference answer: grandson (grandson)

Exact evaluated input as JSON
{
  "state": "[Matthew] likes to watch baseball with his brother. His name is [Anthony]. [Stanley] had a conversation with his grandson, [Matthew]. [Stanley] asked his wife [Molly] to keep her distance.",
  "question": "What is Anthony’s relationship to Molly?",
  "choices": {
    "aunt": "aunt",
    "brother": "brother",
    "daughter": "daughter",
    "daughter-in-law": "daughter-in-law",
    "father": "father",
    "father-in-law": "father-in-law",
    "granddaughter": "granddaughter",
    "grandfather": "grandfather",
    "grandmother": "grandmother",
    "grandson": "grandson",
    "mother": "mother",
    "mother-in-law": "mother-in-law",
    "nephew": "nephew",
    "niece": "niece",
    "sister": "sister",
    "son": "son",
    "son-in-law": "son-in-law",
    "uncle": "uncle"
  }
}

Recorded model answers

Model Answer Outcome
Astra grandson Correct
Opus 5.5 grandson Correct
GLiDE v1.2 grandson Correct
Decider v1.1 grandson Correct
Jev grandson Correct
Microsoft Decision-1 nephew Incorrect
Clef grandson Correct
Luna Decisions son Incorrect
Clef Flash nephew Incorrect
Strands Decider nephew Incorrect
GLiNER2.5 Decide grandson Correct
Laya grandson Correct
CalDec v1 grandson Correct

Source and sharing

Attribution: Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "gen_train23_test2to10/test.csv",
    "record": "74",
    "kind": "source_file_record"
  },
  "source_revision": "e5b496941e91abb7c319d2618a3ce96752bc4ab7",
  "input_hash": "02c8a57a7ef8e3e72a30a352fed9e8924959603fb94eeb71d0487f46160a7ba9"
}

Type to search questions, variants and documentation.