COUNTDOWN
Benchmarks
TaskDatasetSOTA resultTrendResults
Countdown
85Accuracy
252
Countdown
87.5Accuracy
101
Countdown (test)
85Accuracy
84
Countdown
83.2Accuracy
49
Countdown
72.7Pass@1
34
Countdown
63.2Accuracy
31
Countdown
65.6Accuracy
28
Countdown
85.6Accuracy
27
Countdown
63.1Accuracy
25
Countdown 5
56.4RTR (%)
24
Countdown (test)
84.8Accuracy
24
Countdown
49.61Accuracy
24
Countdown 4
86.2RTR (%)
20
Countdown 3
98.4RTR (%)
15
Countdown 512 tokens
62.1Pass@1
15
Countdown 256 tokens
71.1Pass@1
15
Countdown
81.6Accuracy
14
Countdown (held-out)
87.96Pass@1
14
Countdown CD34
78.2Avg@16
14
Countdown CD4
59.4Avg@16
14
Countdown (val)
76.27Validation Reward (mean@4)
13
Countdown-4
98.9CD4
13
Countdown
54.3Accuracy (L=128)
13
COUNTDOWN (test)
66.02Accuracy
13
Countdown 3 (test)
95Reward
12