RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 19

Back to Relations · All capabilities

Question ID: clutrr/2a7af3190a3349b982c2
Source: CLUTRR
Input format: native choices

Explore this question’s altered versions

Context

[Jason] is celebrating his 20th anniversary with his wife [Gabrielle] in Hawaii. [Gabrielle] is so proud of her Grandson, [Robert]. She spoils him rotten.

Question

What is Robert’s relationship to Jason?

Answer choices

  • aunt: aunt
  • brother: brother
  • daughter: daughter
  • daughter-in-law: daughter-in-law
  • father: father
  • father-in-law: father-in-law
  • granddaughter: granddaughter
  • grandfather: grandfather
  • grandmother: grandmother
  • grandson: grandson
  • mother: mother
  • mother-in-law: mother-in-law
  • nephew: nephew
  • niece: niece
  • sister: sister
  • son: son
  • son-in-law: son-in-law
  • uncle: uncle

Reference answer: grandson (grandson)

Exact evaluated input as JSON
{
  "state": "[Jason] is celebrating his 20th anniversary with his wife [Gabrielle] in Hawaii. [Gabrielle] is so proud of her Grandson, [Robert]. She spoils him rotten.",
  "question": "What is Robert’s relationship to Jason?",
  "choices": {
    "aunt": "aunt",
    "brother": "brother",
    "daughter": "daughter",
    "daughter-in-law": "daughter-in-law",
    "father": "father",
    "father-in-law": "father-in-law",
    "granddaughter": "granddaughter",
    "grandfather": "grandfather",
    "grandmother": "grandmother",
    "grandson": "grandson",
    "mother": "mother",
    "mother-in-law": "mother-in-law",
    "nephew": "nephew",
    "niece": "niece",
    "sister": "sister",
    "son": "son",
    "son-in-law": "son-in-law",
    "uncle": "uncle"
  }
}

Recorded model answers

Model Answer Outcome
Astra grandson Correct
Opus 5.5 grandson Correct
GLiDE v1.2 grandson Correct
Decider v1.1 grandson Correct
Jev grandson Correct
Microsoft Decision-1 nephew Incorrect
Clef grandson Correct
Luna Decisions grandson Correct
Clef Flash grandson Correct
Strands Decider grandson Correct
GLiNER2.5 Decide grandson Correct
Laya grandson Correct
CalDec v1 grandson Correct

Source and sharing

Attribution: Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "gen_train23_test2to10/validation.csv",
    "record": "743",
    "kind": "source_file_record"
  },
  "source_revision": "e5b496941e91abb7c319d2618a3ce96752bc4ab7",
  "input_hash": "49d8d72e57b552b2dfab1c5fa1931364bdf0e8f25702d4b444651b800ef7e88b"
}

Type to search questions, variants and documentation.