RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 14

Back to Relations · All capabilities

Question ID: clutrr/924cb5916b7404ac8922
Source: CLUTRR
Input format: native choices

Explore this question’s altered versions

Context

[Tommy] went fishing with his daughter [Judy]. [Judy]'s sister, [Maxine], was n't invited because she hates fishing. [Elizabeth] told a secret to her sister, [Judy]. [Judy] passed it along to her brother, [Reynaldo]. [Martha] went to the store with her sister [Elizabeth] [Maxine] and her son, [Philip], went to the park to fly a kite. They had fun doing it all day. [Reynaldo] was so proud of his son, [Philip]. he received a great scholarship to college.

Question

What is Martha’s relationship to Tommy?

Answer choices

  • aunt: aunt
  • brother: brother
  • daughter: daughter
  • daughter-in-law: daughter-in-law
  • father: father
  • father-in-law: father-in-law
  • granddaughter: granddaughter
  • grandfather: grandfather
  • grandmother: grandmother
  • grandson: grandson
  • mother: mother
  • mother-in-law: mother-in-law
  • nephew: nephew
  • niece: niece
  • sister: sister
  • son: son
  • son-in-law: son-in-law
  • uncle: uncle

Reference answer: daughter (daughter)

Exact evaluated input as JSON
{
  "state": "[Tommy] went fishing with his daughter [Judy]. [Judy]'s sister, [Maxine], was n't invited because she hates fishing. [Elizabeth] told a secret to her sister, [Judy]. [Judy] passed it along to her brother, [Reynaldo]. [Martha] went to the store with her sister [Elizabeth] [Maxine] and her son, [Philip], went to the park to fly a kite. They had fun doing it all day. [Reynaldo] was so proud of his son, [Philip]. he received a great scholarship to college.",
  "question": "What is Martha’s relationship to Tommy?",
  "choices": {
    "aunt": "aunt",
    "brother": "brother",
    "daughter": "daughter",
    "daughter-in-law": "daughter-in-law",
    "father": "father",
    "father-in-law": "father-in-law",
    "granddaughter": "granddaughter",
    "grandfather": "grandfather",
    "grandmother": "grandmother",
    "grandson": "grandson",
    "mother": "mother",
    "mother-in-law": "mother-in-law",
    "nephew": "nephew",
    "niece": "niece",
    "sister": "sister",
    "son": "son",
    "son-in-law": "son-in-law",
    "uncle": "uncle"
  }
}

Recorded model answers

Model Answer Outcome
Astra daughter Correct
Opus 5.5 daughter Correct
GLiDE v1.2 grandmother Incorrect
Decider v1.1 aunt Incorrect
Jev sister Incorrect
Microsoft Decision-1 aunt Incorrect
Clef aunt Incorrect
Luna Decisions aunt Incorrect
Clef Flash aunt Incorrect
Strands Decider sister Incorrect
GLiNER2.5 Decide son Incorrect
Laya daughter Correct
CalDec v1 son Incorrect

Source and sharing

Attribution: Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "gen_train23_test2to10/test.csv",
    "record": "685",
    "kind": "source_file_record"
  },
  "source_revision": "e5b496941e91abb7c319d2618a3ce96752bc4ab7",
  "input_hash": "3ae3eb373cce3e6e2746e45cba83175a20cde4b21a40e2740511c0467470689c"
}

Type to search questions, variants and documentation.