| 1. Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partn… |
BIG-Bench Hard selected tasks |
8/13 |
Included |
| 2. Where is Mary? |
bAbI |
13/13 |
Included |
| 3. Where is the sponge now? |
ToMBench |
13/13 |
Included |
| 4. What does f([35, 45, 3, 61, 39, 27, 47], 0) return? |
CRUXEval |
10/13 |
Included |
| 5. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the sta… |
BIG-Bench Hard selected tasks |
9/13 |
Included |
| 6. Where is the football? |
bAbI |
11/13 |
Included |
| 7. What does f([0, 6, 2, -1, -2]) return? |
CRUXEval |
11/13 |
Included |
| 8. Alice, Bob, Claire, Dave, and Eve are dancers at a square dance. At the start of a song, they each h… |
BIG-Bench Hard selected tasks |
3/13 |
Included |
| 9. Where was the apple before the office? |
bAbI |
7/13 |
Included |
| 10. What does f([3, 4, 1, 2, 3]) return? |
CRUXEval |
10/13 |
Included |
| 11. Where is the football? |
bAbI |
12/13 |
Included |
| 12. Alice, Bob, Claire, Dave, and Eve are holding a white elephant gift exchange. At the start of the ev… |
BIG-Bench Hard selected tasks |
8/13 |
Included |
| 13. What does f([3, 1, 7, 5, 6]) return? |
CRUXEval |
11/13 |
Included |
| 14. Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: A… |
BIG-Bench Hard selected tasks |
6/13 |
Included |
| 15. Where is John? |
bAbI |
12/13 |
Included |
| 16. What does f([1, 3, -1, 1, -2, 6]) return? |
CRUXEval |
11/13 |
Included |
| 17. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the sta… |
BIG-Bench Hard selected tasks |
5/13 |
Included |
| 18. Where was the football before the kitchen? |
bAbI |
7/13 |
Included |
| 19. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… |
BIG-Bench Hard selected tasks |
6/13 |
Included |
| 20. Where is Sandra? |
bAbI |
13/13 |
Included |
| 21. Where is the milk? |
bAbI |
12/13 |
Included |
| 22. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trad… |
BIG-Bench Hard selected tasks |
9/13 |
Included |
| 23. Where was the milk before the garden? |
bAbI |
6/13 |
Included |
| 24. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… |
BIG-Bench Hard selected tasks |
6/13 |
Included |
| 25. Alice, Bob, and Claire are on the same team in a soccer match. At the start of the match, they are e… |
BIG-Bench Hard selected tasks |
7/13 |
Included |
| 26. Where is the apple? |
bAbI |
9/13 |
Included |
| 27. Where is Sandra? |
bAbI |
12/13 |
Included |
| 28. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… |
BIG-Bench Hard selected tasks |
10/13 |
Included |
| 29. Where was the milk before the kitchen? |
bAbI |
6/13 |
Included |
| 30. Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the … |
BIG-Bench Hard selected tasks |
5/13 |
Included |
| 31. Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partn… |
BIG-Bench Hard selected tasks |
7/13 |
Included |
| 32. Where is the apple? |
bAbI |
10/13 |
Included |
| 33. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a s… |
BIG-Bench Hard selected tasks |
9/13 |
Included |
| 34. Where is John? |
bAbI |
13/13 |
Included |
| 35. Where was the apple before the garden? |
bAbI |
6/13 |
Included |
| 36. Alice, Bob, Claire, Dave, and Eve are friends and avid readers who occasionally trade books. At the … |
BIG-Bench Hard selected tasks |
7/13 |
Included |
| 37. Alice, Bob, and Claire are on the same team in a soccer match. At the start of the match, they are e… |
BIG-Bench Hard selected tasks |
10/13 |
Included |
| 38. Where is the milk? |
bAbI |
11/13 |
Included |
| 39. Where is John? |
bAbI |
12/13 |
Included |
| 40. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a s… |
BIG-Bench Hard selected tasks |
4/13 |
Included |
| 41. Where was the apple before the office? |
bAbI |
10/13 |
Included |
| 42. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… |
BIG-Bench Hard selected tasks |
10/13 |
Included |
| 43. Where is Mary? |
bAbI |
12/13 |
Included |
| 44. Alice, Bob, and Claire are friends and avid readers who occasionally trade books. At the start of th… |
BIG-Bench Hard selected tasks |
7/13 |
Included |
| 45. Where was the apple before the hallway? |
bAbI |
10/13 |
Included |
| 46. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the… |
BIG-Bench Hard selected tasks |
5/13 |
Included |
| 47. Where is the apple? |
bAbI |
5/13 |
Included |
| 48. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… |
BIG-Bench Hard selected tasks |
5/13 |
Included |
| 49. Where is John? |
bAbI |
11/13 |
Included |
| 50. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… |
BIG-Bench Hard selected tasks |
8/13 |
Included |
| 51. Where is the milk? |
bAbI |
11/13 |
Included |
| 52. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… |
BIG-Bench Hard selected tasks |
7/13 |
Included |
| 53. Where was the milk before the hallway? |
bAbI |
7/13 |
Included |
| 54. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… |
BIG-Bench Hard selected tasks |
5/13 |
Included |
| 55. Where is Sandra? |
bAbI |
10/13 |
Included |
| 56. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… |
BIG-Bench Hard selected tasks |
9/13 |
Included |
| 57. Where is the milk? |
bAbI |
9/13 |
Included |
| 58. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trad… |
BIG-Bench Hard selected tasks |
6/13 |
Included |
| 59. Where was the apple before the bathroom? |
bAbI |
6/13 |
Included |
| 60. Alice, Bob, Claire, Dave, and Eve are playing a game. At the start of the game, they are each holdin… |
BIG-Bench Hard selected tasks |
10/13 |
Included |
| 61. Where is John? |
bAbI |
11/13 |
Included |
| 62. Alice, Bob, and Claire are holding a white elephant gift exchange. At the start of the event, they a… |
BIG-Bench Hard selected tasks |
8/13 |
Included |
| 63. Where is the football? |
bAbI |
11/13 |
Included |
| 64. Alice, Bob, Claire, Dave, and Eve are on the same team in a soccer match. At the start of the match,… |
BIG-Bench Hard selected tasks |
6/13 |
Included |
| 65. Where was the milk before the hallway? |
bAbI |
3/13 |
Included |
| 66. Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they… |
BIG-Bench Hard selected tasks |
4/13 |
Included |
| 67. Where is Mary? |
bAbI |
11/13 |
Included |