RaindropA practical guide to choosing decision modelsGitHub ↗

Rule application

All capabilities · Methods

67 questions. ‘Correct / answered’ counts the recorded outcomes across all 13 models.

Question Source Correct / answered Question text
1. Which response better satisfies the request? RewardBench 10/13 Included
2. Recover locally — depth-1 ProofWriter 7/13 Recover locally
3. What does f('staovk', 'k') return? CRUXEval 10/13 Included
4. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 10/13 Included
5. Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench 12/13 Included
6. Select the best English interpretation of the given proposition, using the following translation ke… MMLU 11/13 Included
7. Recover locally — depth-2 ProofWriter 10/13 Recover locally
8. Which response better satisfies the request? RewardBench 13/13 Included
9. S(n) is a statement about positive integers n such that whenever S(k) is true, S(k + 1) must also be… MMLU 7/13 Included
10. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 7/13 Included
11. What does f("") return? CRUXEval 13/13 Included
12. Recover locally — depth-6 ProofWriter 4/13 Recover locally
13. Recover locally — depth-3 ProofWriter 7/13 Recover locally
14. Which response better satisfies the request? RewardBench 12/13 Included
15. Construct a complete truth table for the following argument. Then, using the truth table, determine… MMLU 7/13 Included
16. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
17. What does f("odes\tcode\twell", 2) return? CRUXEval 10/13 Included
18. Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench 11/13 Included
19. Which response better satisfies the request? RewardBench 9/13 Included
20. Recover locally — depth-4 ProofWriter 8/13 Recover locally
21. Use indirect truth tables to determine whether the following argument is valid. If the argument is i… MMLU 12/13 Included
22. What does f('ax') return? CRUXEval 7/13 Included
23. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 13/13 Included
24. Recover locally — depth-4 ProofWriter 6/13 Recover locally
25. Which response better satisfies the request? RewardBench 12/13 Included
26. Recover locally — depth-0 ProofWriter 8/13 Recover locally
27. What does f('ncdsdfdaaa0a1cdscsk*XFd') return? CRUXEval 12/13 Included
28. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 9/13 Included
29. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 8/13 Included
30. Which response better satisfies the request? RewardBench 9/12 Included
31. Recover locally — depth-6 ProofWriter 9/13 Recover locally
32. What does f('the speed is -36 miles per hour') return? CRUXEval 12/13 Included
33. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
34. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 10/13 Included
35. Recover locally — depth-5 ProofWriter 8/13 Recover locally
36. Which response better satisfies the request? RewardBench 10/13 Included
37. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
38. What does f('abc', 'abcd') return? CRUXEval 9/13 Included
39. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 10/13 Included
40. Recover locally — depth-7 ProofWriter 9/13 Recover locally
41. Which response better satisfies the request? RewardBench 12/13 Included
42. What does f([4, 5]) return? CRUXEval 8/13 Included
43. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
44. Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench 9/13 Included
45. Recover locally — depth-2 ProofWriter 9/13 Recover locally
46. What does f('pvtso') return? CRUXEval 13/13 Included
47. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
48. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 10/13 Included
49. Recover locally — depth-3 ProofWriter 5/13 Recover locally
50. What does f(range(100, 120)) return? CRUXEval 11/13 Included
51. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 11/13 Included
52. Recover locally — depth-2 ProofWriter 9/13 Recover locally
53. Recover locally — depth-1 ProofWriter 12/13 Recover locally
54. What does f('abc', 1) return? CRUXEval 9/13 Included
55. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 9/13 Included
56. Does the target assistant step contain an error under a process-quality evaluation? A correct useful… AgentProcessBench 11/13 Included
57. Recover locally — depth-4 ProofWriter 6/13 Recover locally
58. What does f('K zBK') return? CRUXEval 10/13 Included
59. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 10/13 Included
60. Recover locally — depth-3 ProofWriter 7/13 Recover locally
61. Recover locally — depth-5 ProofWriter 6/13 Recover locally
62. What does f('m4n2o') return? CRUXEval 10/13 Included
63. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 12/13 Included
64. Recover locally — depth-6 ProofWriter 5/13 Recover locally
65. What does f({'-4':'4','1':'2','-':'-3'}) return? CRUXEval 12/13 Included
66. Which candidate response is most appropriate, given the user's request and the available tools? When2Call 9/13 Included
67. Recover locally — depth-1 ProofWriter 12/13 Recover locally

Type to search questions, variants and documentation.