Reasoning and Code Generation on Reasoning Evaluation Suite (GSM8K, MATH500, AIME24, HumanEval) (test)
94.7GSM8K AccuracySafeChain
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SafeChainBackbone=s1-14B, Dataset Size=1K, Temperature=0.02026.04 | 94.7 | 84.6 | 26.7 | 66.5 | 68.1 | |
| No trainBackbone=s1-14B, Dataset Size=-, Temperature=0.02026.04 | 93.7 | 80.4 | 20 | 69.5 | 65.9 | |
| STAR-1Backbone=R1-32B, Dataset Size=1K, Temperature=0.02026.04 | 92.9 | 88.6 | 56.7 | 79.9 | 79.5 | |
| S1-ALTBackbone=s1-14B, Dataset Size=1K, Temperature=0.02026.04 | 92.7 | 80.6 | 20 | 65.8 | 64.8 | |
| SafeChainBackbone=R1-32B, Dataset Size=1K, Temperature=0.02026.04 | 92.1 | 83.8 | 50 | 81.7 | 76.9 | |
| R1-ALTBackbone=R1-32B, Dataset Size=1K, Temperature=0.02026.04 | 91.4 | 84.2 | 53.3 | 82.9 | 78 | |
| No trainBackbone=R1-32B, Dataset Size=-, Temperature=0.02026.04 | 91.2 | 85.6 | 46.7 | 80.5 | 76 | |
| STAR-1Backbone=R1-14B, Dataset Size=1K, Temperature=0.02026.04 | 90.9 | 84.8 | 40 | 83.5 | 74.8 | |
| SafeChainBackbone=R1-32B, Dataset Size=40K, Temperature=0.02026.04 | 90.8 | 83.8 | 60 | 82.3 | 79.2 | |
| DirectRefusalBackbone=R1-32B, Dataset Size=1k, Temperature=0.02026.04 | 90.4 | 80.8 | 46.7 | 84.8 | 75.7 | |
| No trainBackbone=R1-14B, Dataset Size=-, Temperature=0.02026.04 | 89.9 | 84 | 40 | 83.5 | 74.4 | |
| DirectRefusalBackbone=R1-14B, Dataset Size=1k, Temperature=0.02026.04 | 89.8 | 80 | 36.7 | 81.1 | 71.9 | |
| SafeChainBackbone=R1-14B, Dataset Size=1K, Temperature=0.02026.04 | 89.2 | 83 | 40 | 82.3 | 73.6 | |
| SafeChainBackbone=R1-14B, Dataset Size=40K, Temperature=0.02026.04 | 89.1 | 83 | 36.7 | 81.7 | 72.6 | |
| R1-ALTBackbone=R1-14B, Dataset Size=1K, Temperature=0.02026.04 | 88.6 | 84.8 | 40 | 84.8 | 74.6 | |
| R1-ALTBackbone=R1-7B, Dataset Size=1K, Temperature=0.02026.04 | 86.6 | 84.6 | 36.7 | 70.1 | 69.5 | |
| SafeChainBackbone=R1-7B, Dataset Size=40K, Temperature=0.02026.04 | 86 | 80.6 | 16.7 | 64.6 | 62 | |
| No trainBackbone=R1-7B, Dataset Size=-, Temperature=0.02026.04 | 85.1 | 84.6 | 43.3 | 77.4 | 72.6 | |
| SafeChainBackbone=R1-7B, Dataset Size=1K, Temperature=0.02026.04 | 85.1 | 84.4 | 30 | 68.9 | 67.1 | |
| STAR-1Backbone=R1-7B, Dataset Size=1K, Temperature=0.02026.04 | 85.1 | 85.6 | 36.7 | 77.4 | 71.2 | |
| DirectRefusalBackbone=R1-7B, Dataset Size=1k, Temperature=0.02026.04 | 80 | 75.4 | 36.7 | 31.1 | 55.8 | |
| S1-ALTBackbone=s1-3B, Dataset Size=1K, Temperature=0.02026.04 | 79.8 | 51 | 0 | 38.4 | 42.3 | |
| No trainBackbone=s1-3B, Dataset Size=-, Temperature=0.02026.04 | 78.6 | 53.2 | 0 | 29.9 | 40.4 | |
| SafeChainBackbone=s1-3B, Dataset Size=1K, Temperature=0.02026.04 | 77.7 | 49.4 | 0 | 39.6 | 41.7 | |
| SafeChainBackbone=R1-8B, Dataset Size=40K, Temperature=0.02026.04 | 72 | 71.6 | 16.7 | 66.5 | 56.7 | |
| SafeChainBackbone=R1-8B, Dataset Size=1K, Temperature=0.02026.04 | 70.7 | 76.6 | 30 | 67.1 | 61.1 | |
| No trainBackbone=R1-8B, Dataset Size=-, Temperature=0.02026.04 | 70.2 | 72.4 | 23.3 | 66.5 | 58.1 | |
| STAR-1Backbone=R1-8B, Dataset Size=1K, Temperature=0.02026.04 | 69.6 | 69.8 | 16.7 | 67.7 | 56 | |
| R1-ALTBackbone=R1-8B, Dataset Size=1K, Temperature=0.02026.04 | 69 | 74.4 | 26.7 | 68.9 | 59.8 | |
| DirectRefusalBackbone=R1-8B, Dataset Size=1k, Temperature=0.02026.04 | 66.9 | 65 | 20 | 64.6 | 54.1 | |
| SafeChainBackbone=R1-1.5B, Dataset Size=40k, Temperature=0.02026.04 | 51.4 | 45.2 | 0 | 43.9 | 35.1 | |
| No trainBackbone=R1-1.5B, Dataset Size=-, Temperature=0.02026.04 | 50.3 | 44.6 | 6.7 | 42.7 | 36.1 | |
| SafeChainBackbone=R1-1.5B, Dataset Size=1K, Temperature=0.02026.04 | 49.7 | 46 | 0 | 45.7 | 35.4 | |
| R1-ALTBackbone=R1-1.5B, Dataset Size=1K, Temperature=0.02026.04 | 49.4 | 43.6 | 13.3 | 39 | 36.3 | |
| DirectRefusalBackbone=R1-1.5B, Dataset Size=1k, Temperature=0.02026.04 | 49.3 | 45.2 | 3.3 | 43.9 | 35.4 | |
| STAR-1Backbone=R1-1.5B, Dataset Size=1K, Temperature=0.02026.04 | 45 | 51.2 | 10 | 53.7 | 40 |