RaindropA practical guide to choosing decision modelsGitHub ↗

Relations · question 44

Back to Relations · All capabilities

Question ID: clutrr/82c94d7e363144fc2af8
Source: CLUTRR
Input format: native choices

Explore this question’s altered versions

Context

[Asia] is a proud sibling of the amazing [Russell]. [Russell] has mixed feelings towards his mother, [Alma].

Question

What is Alma’s relationship to Asia?

Answer choices

  • aunt: aunt
  • brother: brother
  • daughter: daughter
  • daughter-in-law: daughter-in-law
  • father: father
  • father-in-law: father-in-law
  • granddaughter: granddaughter
  • grandfather: grandfather
  • grandmother: grandmother
  • grandson: grandson
  • mother: mother
  • mother-in-law: mother-in-law
  • nephew: nephew
  • niece: niece
  • sister: sister
  • son: son
  • son-in-law: son-in-law
  • uncle: uncle

Reference answer: mother (mother)

Exact evaluated input as JSON
{
  "state": "[Asia] is a proud sibling of the amazing [Russell]. [Russell] has mixed feelings towards his mother, [Alma].",
  "question": "What is Alma’s relationship to Asia?",
  "choices": {
    "aunt": "aunt",
    "brother": "brother",
    "daughter": "daughter",
    "daughter-in-law": "daughter-in-law",
    "father": "father",
    "father-in-law": "father-in-law",
    "granddaughter": "granddaughter",
    "grandfather": "grandfather",
    "grandmother": "grandmother",
    "grandson": "grandson",
    "mother": "mother",
    "mother-in-law": "mother-in-law",
    "nephew": "nephew",
    "niece": "niece",
    "sister": "sister",
    "son": "son",
    "son-in-law": "son-in-law",
    "uncle": "uncle"
  }
}

Recorded model answers

Model Answer Outcome
Astra mother Correct
Opus 5.5 mother Correct
GLiDE v1.2 mother Correct
Decider v1.1 mother Correct
Jev mother Correct
Microsoft Decision-1 mother Correct
Clef mother Correct
Luna Decisions mother Correct
Clef Flash mother Correct
Strands Decider sister Incorrect
GLiNER2.5 Decide brother Incorrect
Laya mother Correct
CalDec v1 mother Correct

Source and sharing

Attribution: Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton.
Terms: CC-BY-NC-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "gen_train23_test2to10/train.csv",
    "record": "1522",
    "kind": "source_file_record"
  },
  "source_revision": "e5b496941e91abb7c319d2618a3ce96752bc4ab7",
  "input_hash": "7aa05987e3788163fcec8d26856aa1de92debc8ffe696225845b97a2149ccd46"
}

Type to search questions, variants and documentation.