RaindropA practical guide to choosing decision modelsGitHub ↗

Binary subset

All 1,005 questions · Binary results · Scoring · Download question IDs

These 214 questions from the 1,005-question set supply the post's precision, recall, false-alarm and missed-detection results. Each has two answer choices and an explicit positive label from its source. Fifteen use adapted choice representations.

Each question page shows its source, reference answer and all 13 recorded model outcomes. Positive choice is the choice key treated as positive in scoring; its meaning varies by source. ‘Correct / answered’ counts saved answers across all 13 models. Questions whose text is obtained separately remain listed with their results and recovery instructions.

Retrieval · 16 questions

Question Source Positive choice Correct / answered
is there a new jungle book movie coming out BoolQ yes 13/13
do you get paid when on administrative leave BoolQ yes 13/13
did a helicopter crash in the perfect storm BoolQ yes 13/13
can a blighted ovum have a yolk sac BoolQ yes 12/13
was the sears tower ever the tallest building in the world BoolQ yes 13/13
does deadpool have a kid in the comics BoolQ yes 13/13
did they film the lord of the rings all at once BoolQ yes 10/13
roswell do max and liz end up together BoolQ yes 13/13
does superman appear in the movie justice league BoolQ yes 13/13
is a swift code the same as a bsb number BoolQ yes 12/13
is a group of cats known as a clowder BoolQ yes 11/13
was this the last season of blue bloods BoolQ yes 12/13
is king louie in the jungle book book BoolQ yes 13/13
did the atlanta falcons get a new stadium BoolQ yes 13/13
are the channel islands a part of great britain BoolQ yes 13/13
is there a sequel to batman versus superman BoolQ yes 13/13

Concepts and categorization · 18 questions

Question Source Positive choice Correct / answered
Does the assistant response refuse the user's request? WildGuardTest refusal 13/13
Recover locally — sentiment SST-2 1 12/13
Does this user message contain toxic content? ToxicChat 1 13/13
Is the user's request harmful? WildGuardTest harmful 13/13
Recover locally — sentiment SST-2 1 13/13
Does this user message contain toxic content? ToxicChat 1 13/13
Does the assistant response refuse the user's request? WildGuardTest refusal 13/13
Recover locally — sentiment SST-2 1 12/13
Does this user message contain toxic content? ToxicChat 1 13/13
Is the user's request harmful? WildGuardTest harmful 8/13
Recover locally — sentiment SST-2 1 13/13
Does this user message contain toxic content? ToxicChat 1 13/13
Does the assistant response refuse the user's request? WildGuardTest refusal 13/13
Recover locally — sentiment SST-2 1 13/13
Does this user message contain toxic content? ToxicChat 1 13/13
Does the assistant response contain harmful content? WildGuardTest harmful 10/13
Recover locally — sentiment SST-2 1 13/13
Does this user message contain toxic content? ToxicChat 1 13/13

Physical cognition · 3 questions

Question Source Positive choice Correct / answered
Recover locally — boolean NEWTON 1 7/13
Recover locally — boolean NEWTON 1 10/13
Recover locally — boolean NEWTON 1 12/13

Spatial cognition · 15 questions

Question Source Positive choice Correct / answered
Is the red square above the pink rectangle? bAbI yes 12/13
Is the triangle above the blue square? bAbI yes 11/13
Is the yellow square below the triangle? bAbI yes 12/13
Is the pink rectangle below the red square? bAbI yes 12/13
Is the red square below the yellow square? bAbI yes 12/13
Is the triangle above the blue square? bAbI yes 11/13
Is the triangle to the right of the pink rectangle? bAbI yes 11/13
Is the red square below the red sphere? bAbI yes 11/13
Is the triangle above the red square? bAbI yes 11/13
Is the yellow square to the left of the pink rectangle? bAbI yes 11/13
Is the red sphere above the yellow square? bAbI yes 10/13
Is the yellow square below the triangle? bAbI yes 11/13
Is the blue square to the left of the red square? bAbI yes 10/13
Is the yellow square below the red sphere? bAbI yes 12/13
Is the red square above the red sphere? bAbI yes 10/13

Numerical cognition · 15 questions

Question Source Positive choice Correct / answered
Is smallpox survival less likely than smallpox death overall? CLadder yes 8/13
Is the chance of black hair larger when observing tanning salon treatment? CLadder yes 7/13
Is glimx more likely than not glimx overall? CLadder yes 11/13
Is the chance of kwox larger when observing jyka? CLadder yes 10/13
Is rukz less likely than not rukz overall? CLadder yes 11/13
Is rukz less likely than not rukz overall? CLadder yes 11/13
Is muvq more likely than not muvq overall? CLadder yes 13/13
Is xevu less likely than not xevu overall? CLadder yes 8/13
Is xevu more likely than not xevu overall? CLadder yes 11/13
Is lirg more likely than not lirg overall? CLadder yes 12/13
Is uvzi less likely than not uvzi overall? CLadder yes 9/13
Is kwox more likely than not kwox overall? CLadder yes 11/13
Is uvzi less likely than not uvzi overall? CLadder yes 7/13
Is uvzi more likely than not uvzi overall? CLadder yes 9/13
Is kwox less likely than not kwox overall? CLadder yes 12/13

Causal reasoning · 57 questions

Question Source Positive choice Correct / answered
For those who are rixq, would it be less likely to see xevu if the individual was not rixq? CLadder yes 11/13
To understand how jyka affects kwox, is it more correct to use the Method 1 than Method 2? CLadder yes 9/13
Will yupt decrease the chance of muvq? CLadder yes 12/13
If we look at people who are famous, does it mean that attractive appearance affects talent? CLadder yes 12/13
If we disregard the mediation effect through swoq, would kwox negatively affect kwoz? CLadder yes 11/13
Would an individual is not xevu if not rixq instead of rixq? CLadder yes 9/13
Does yomx positively affect xevu through gwet? CLadder yes 11/13
For those who are jyka, would it be less likely to see lirg if the individual was not jyka? CLadder yes 11/13
To understand how jyka affects kwox, is it more correct to use the Method 1 than Method 2? CLadder yes 9/13
If we look at people in a relationship, does it mean that kindness affects attractive appearance? CLadder yes 12/13
If we disregard the mediation effect through tijv, would xevo positively affect gyzp? CLadder yes 11/13
Would Alice arrives at school late if waking up late instead of waking up on time? CLadder yes 11/13
Does rixq negatively affect xevu through zuph? CLadder yes 8/13
Will kwox decrease the chance of kwoz? CLadder yes 10/13
For those who are zuph, would it be less likely to see uvzi if the individual was not zuph? CLadder yes 10/13
If we look at hospitalized individuals, does it mean that respiratory issues affects broken bones? CLadder yes 12/13
Would an individual is uvzi if zuph instead of not zuph? CLadder yes 11/13
To understand how kwox affects kwoz, is it more correct to use the Method 1 than Method 2? CLadder yes 7/13
Does pexu negatively affect rukz through hwax? CLadder yes 11/13
Will full moon increase the chance of wet ground? CLadder yes 12/13
If we disregard the mediation effect through wife, would drinking coffee negatively affect alarm clo… CLadder yes 11/13
For those who are zuph, would it be more likely to see glimx if the individual was not zuph? CLadder yes 11/13
If we look at those who are yupt, does it mean that jyka affects kwox? CLadder yes 12/13
Would an individual is kwox if not jyka instead of jyka? CLadder yes 10/13
Does education level positively affect salary through skill? CLadder yes 13/13
Will jyka decrease the chance of kwox? CLadder yes 9/13
To understand how zuph affects glimx, is it more correct to use the Method 1 than Method 2? CLadder yes 8/13
If we disregard the mediation effect through zuph, would rixq positively affect xevu? CLadder yes 11/13
For those who are rixq, would it be less likely to see xevu if the individual was not rixq? CLadder yes 11/13
Would the child is intelligent if not liking spicy food instead of liking spicy food? CLadder yes 9/13
Does smoking positively affect lactose intolerance through effort? CLadder yes 9/13
If we look at students accepted to elite institutions, does it mean that listening to jazz affects b… CLadder yes 13/13
To understand how pexu affects rukz, is it more correct to use the Method 1 than Method 2? CLadder yes 8/13
Will xevo decrease the chance of gyzp? CLadder yes 11/13
If we disregard the mediation effect through zupj, would yupt positively affect muvq? CLadder yes 11/13
For those who are xevo, would it be less likely to see gyzp if the individual was not xevo? CLadder yes 10/13
Does pexu negatively affect rukz through hwax and kraz? CLadder yes 12/13
Would an individual is not lirg if jyka instead of not jyka? CLadder yes 13/13
Will male gender increase the chance of admission acceptance? CLadder yes 12/13
If we look at those who are gwet, does it mean that yomx affects xevu? CLadder yes 12/13
To understand how jyka affects kwox, is it more correct to use the Method 1 than Method 2? CLadder yes 8/13
For captains who execute prisoners, would it be less likely to see the prisoner's death if the capta… CLadder yes 10/13
Does jyka positively affect kwox through yupt and qwiu? CLadder yes 12/13
Will citrus intake decrease the chance of scurvy? CLadder yes 13/13
Would an individual is uvzi if not zuph instead of zuph? CLadder yes 11/13
If we look at hospitalized individuals, does it mean that respiratory issues affects broken bones? CLadder yes 11/13
To understand how pexu affects rukz, is it more correct to use the Method 1 than Method 2? CLadder yes 10/13
For people who went to tanning salons, would it be more likely to see black hair if the person had n… CLadder yes 10/13
Does listening to jazz positively affect lung cancer through tar deposit? CLadder yes 13/13
Will encouragement decrease the chance of high exam score? CLadder yes 12/13
Would an individual is not glimx if not zuph instead of zuph? CLadder yes 10/13
If we look at those who are yupt, does it mean that jyka affects kwox? CLadder yes 12/13
For those who are xevo, would it be less likely to see gyzp if the individual was not xevo? CLadder yes 11/13
Does pexu positively affect rukz through hwax? CLadder yes 9/13
Would an individual is muvq if yupt instead of not yupt? CLadder yes 11/13
Will tanning salon treatment decrease the chance of tanned skin? CLadder yes 12/13
If we look at those who are zupj, does it mean that yupt affects muvq? CLadder yes 12/13

Uncertainty monitoring · 57 questions

Question Source Positive choice Correct / answered
If revenge is a dish that’s best served cold, and revenge is sweet, then is revenge ice cream? SelfAware no 7/13
do you know the place where the peacock throne is now BoolQ yes 11/13
What is a dog's responses to stimuli known as? SelfAware no 11/13
"""f you wanna make the world a better place, take a look at yourself then make a change"" is a line… SelfAware no 10/13
On what day of creation did God make the Sun, the Moon and the Stars? SelfAware no 11/13
What is the equality of forces between two objects exerting force on each other?? SelfAware no 13/13
Conjoined twins are also known as what? SelfAware no 10/13
Nancy's old washing machine could only wash 9 pieces of clothing at a time. If she had to wash 19 sh… SelfAware no 10/13
When did the Tertiary period begin? SelfAware no 12/13
Which primary colour is mixed with white to produce pink? SelfAware no 10/13
How much money is there in the world? SelfAware no 8/13
At 51° 23' E, what is the easternmost point of the African continent? SelfAware no 11/13
Where do all of those who have died go after they die? SelfAware no 12/13
For radio broadcast that one would hear in their car what type of towers are used? SelfAware no 10/13
The diet of what mythical monster periodically included seven youths and seven maidens? SelfAware no 11/13
"Billy Blaze, an 8-year-old genius who constructed a spaceship in his backyard from old soup cans an… SelfAware no 9/13
What restaurant chain with over 15,000 locations is a sister franchise to Wing Street? SelfAware no 10/13
What substance are feathers, fingernails and hair made of? SelfAware no 11/13
Which monthly magazine is based on food entertainment, Club or Food Network Magazine? SelfAware no 11/13
What are teeth made out of? SelfAware no 11/13
Besides the moon, what is the brightest object in the night sky? SelfAware no 13/13
What is karma and how does it operate within us during different lifetimes? SelfAware no 6/13
Who did Sirhan Sirhan assassinate? SelfAware no 12/13
Carol and her mom were picking carrots from their garden. Carol picked 29 and her mother picked 16. … SelfAware no 9/13
Are adolescents or children better able to understand that people do not have complete control over … SelfAware no 10/13
If we knew what created the bing bang, do you think we would try to do it again? SelfAware no 12/13
Which country has the highest nominal wealth per adult in the world according to Credit Suisse? SelfAware no 13/13
In deism is God represented in human form? SelfAware no 10/13
Who was right about reincarnation: Was Buddha right about reincarnation and Hindus were wrong, or vi… SelfAware no 11/13
When we forget, where does the information go? SelfAware no 11/13
What is known as artificial light that is excessive or intrusive? SelfAware no 10/13
Do you think a singularity will happen in this century? SelfAware no 11/13
What is our place in an infinite universe? SelfAware no 12/13
Are lies better than harsh truths? SelfAware no 11/13
What incident killed Diana in Paris? SelfAware no 9/13
Is a turtle homeless or naked, without a shell? SelfAware no 6/13
Would time continue if everything in the world was frozen? SelfAware no 9/13
Stonehenge is in which UK county? SelfAware no 10/13
Where does the lost time go to? SelfAware no 11/13
What is the nautical term for a staircase between two decks on a ship? SelfAware no 11/13
If data scientists had the ability to accurately predict who was more likely to commit crimes in the… SelfAware no 10/13
In the Bible who was the son of Abraham and Sarah? SelfAware no 11/13
Will the advancement of today’s technologies yield a net positive or a net negative result? SelfAware no 11/13
What type of sleep involves dreaming? SelfAware no 10/13
Walter Pauk created the note-taking system devised in what decade? SelfAware no 10/13
Which large fleshy fruit of the far east has been banned from some public areas because of it's dist… SelfAware no 12/13
How many strings are there on a Viola? SelfAware no 11/13
Would you rather have to wear a bib every time you go out to eat or drink from a sippy cup every tim… SelfAware no 10/13
If you were a nyctophobic, what would you fear? SelfAware no 11/13
How long will people remember you after you die? SelfAware no 11/13
What eight letter word do americans use to call what we British call a dressing gown? SelfAware no 12/13
Is the Higgs-Boson machine a good or bad thing for our universe? SelfAware no 10/13
Who is a senior servant in a large household, specifically in charge of the dining room, wine cellar… SelfAware no 11/13
Who invented the art of milking a cow? SelfAware no 11/13
How do vampires dress so fancy if they can’t see their reflections? SelfAware no 11/13
How many undiscovered animals are there in the world? SelfAware no 11/13
In Australia, on which side of the road do motorists drive? SelfAware no 10/13

Communication · 29 questions

Question Source Positive choice Correct / answered
Is this message intended to be sarcastic? iSarcasmEval yes 9/13
Is this message intended to be sarcastic? iSarcasmEval yes 10/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 6/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 8/13
Is this message intended to be sarcastic? iSarcasmEval yes 8/13
Is this message intended to be sarcastic? iSarcasmEval yes 12/13
Is this message intended to be sarcastic? iSarcasmEval yes 12/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 11/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 11/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 8/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 9/13
Is this message intended to be sarcastic? iSarcasmEval yes 7/13
Is this message intended to be sarcastic? iSarcasmEval yes 9/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 8/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 12/13
Is this message intended to be sarcastic? iSarcasmEval yes 10/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13
Is this message intended to be sarcastic? iSarcasmEval yes 13/13

Rule application · 4 questions

Question Source Positive choice Correct / answered
Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench yes 12/13
Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench yes 11/13
Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench yes 9/13
Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench yes 11/13

Type to search questions, variants and documentation.