RaindropA practical guide to choosing decision modelsGitHub ↗

State tracking and updating

All capabilities · Methods

67 questions. ‘Correct / answered’ counts the recorded outcomes across all 13 models.

Question Source Correct / answered Question text
1. Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partn… BIG-Bench Hard selected tasks 8/13 Included
2. Where is Mary? bAbI 13/13 Included
3. Where is the sponge now? ToMBench 13/13 Included
4. What does f([35, 45, 3, 61, 39, 27, 47], 0) return? CRUXEval 10/13 Included
5. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the sta… BIG-Bench Hard selected tasks 9/13 Included
6. Where is the football? bAbI 11/13 Included
7. What does f([0, 6, 2, -1, -2]) return? CRUXEval 11/13 Included
8. Alice, Bob, Claire, Dave, and Eve are dancers at a square dance. At the start of a song, they each h… BIG-Bench Hard selected tasks 3/13 Included
9. Where was the apple before the office? bAbI 7/13 Included
10. What does f([3, 4, 1, 2, 3]) return? CRUXEval 10/13 Included
11. Where is the football? bAbI 12/13 Included
12. Alice, Bob, Claire, Dave, and Eve are holding a white elephant gift exchange. At the start of the ev… BIG-Bench Hard selected tasks 8/13 Included
13. What does f([3, 1, 7, 5, 6]) return? CRUXEval 11/13 Included
14. Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: A… BIG-Bench Hard selected tasks 6/13 Included
15. Where is John? bAbI 12/13 Included
16. What does f([1, 3, -1, 1, -2, 6]) return? CRUXEval 11/13 Included
17. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the sta… BIG-Bench Hard selected tasks 5/13 Included
18. Where was the football before the kitchen? bAbI 7/13 Included
19. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… BIG-Bench Hard selected tasks 6/13 Included
20. Where is Sandra? bAbI 13/13 Included
21. Where is the milk? bAbI 12/13 Included
22. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trad… BIG-Bench Hard selected tasks 9/13 Included
23. Where was the milk before the garden? bAbI 6/13 Included
24. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… BIG-Bench Hard selected tasks 6/13 Included
25. Alice, Bob, and Claire are on the same team in a soccer match. At the start of the match, they are e… BIG-Bench Hard selected tasks 7/13 Included
26. Where is the apple? bAbI 9/13 Included
27. Where is Sandra? bAbI 12/13 Included
28. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… BIG-Bench Hard selected tasks 10/13 Included
29. Where was the milk before the kitchen? bAbI 6/13 Included
30. Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the … BIG-Bench Hard selected tasks 5/13 Included
31. Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partn… BIG-Bench Hard selected tasks 7/13 Included
32. Where is the apple? bAbI 10/13 Included
33. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a s… BIG-Bench Hard selected tasks 9/13 Included
34. Where is John? bAbI 13/13 Included
35. Where was the apple before the garden? bAbI 6/13 Included
36. Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the … BIG-Bench Hard selected tasks 7/13 Included
37. Alice, Bob, and Claire are on the same team in a soccer match. At the start of the match, they are e… BIG-Bench Hard selected tasks 10/13 Included
38. Where is the milk? bAbI 11/13 Included
39. Where is John? bAbI 12/13 Included
40. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a s… BIG-Bench Hard selected tasks 4/13 Included
41. Where was the apple before the office? bAbI 10/13 Included
42. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… BIG-Bench Hard selected tasks 10/13 Included
43. Where is Mary? bAbI 12/13 Included
44. Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of th… BIG-Bench Hard selected tasks 7/13 Included
45. Where was the apple before the hallway? bAbI 10/13 Included
46. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the… BIG-Bench Hard selected tasks 5/13 Included
47. Where is the apple? bAbI 5/13 Included
48. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… BIG-Bench Hard selected tasks 5/13 Included
49. Where is John? bAbI 11/13 Included
50. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… BIG-Bench Hard selected tasks 8/13 Included
51. Where is the milk? bAbI 11/13 Included
52. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… BIG-Bench Hard selected tasks 7/13 Included
53. Where was the milk before the hallway? bAbI 7/13 Included
54. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… BIG-Bench Hard selected tasks 5/13 Included
55. Where is Sandra? bAbI 10/13 Included
56. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… BIG-Bench Hard selected tasks 9/13 Included
57. Where is the milk? bAbI 9/13 Included
58. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trad… BIG-Bench Hard selected tasks 6/13 Included
59. Where was the apple before the bathroom? bAbI 6/13 Included
60. Alice, Bob, Claire, Dave, and Eve are playing a game. At the start of the game, they are each holdin… BIG-Bench Hard selected tasks 10/13 Included
61. Where is John? bAbI 11/13 Included
62. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… BIG-Bench Hard selected tasks 8/13 Included
63. Where is the football? bAbI 11/13 Included
64. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… BIG-Bench Hard selected tasks 6/13 Included
65. Where was the milk before the hallway? bAbI 3/13 Included
66. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… BIG-Bench Hard selected tasks 4/13 Included
67. Where is Mary? bAbI 11/13 Included

Type to search questions, variants and documentation.