Scientific Reasoning on GPQA (test) (Accuracy and Length Metrics)
0.609AccuracyCAT
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CATBackbone=Qwen3-8B2026.07 | 0.609 | 4,819 | 4,393 | 0.334 | 0.309 | |
| Qwen3-8BBackbone=Qwen3-8B2026.07 | 0.599 | 7,241 | 6,354 | — | — | |
| DASTBackbone=Qwen3-8B2026.07 | 0.599 | 6,380 | 5,523 | 0.119 | 0.131 | |
| ConCISESimPOBackbone=Qwen3-8B2026.07 | 0.586 | 3,939 | 3,464 | 0.456 | 0.455 | |
| OverThinkSimPOBackbone=Qwen3-8B2026.07 | 0.571 | 3,712 | 3,389 | 0.487 | 0.467 | |
| CATBackbone=DeepSeek-R1-Distill-Qwen-7B2026.07 | 0.54 | 7,781 | 6,425 | 0.09 | 0.13 | |
| DASTBackbone=DeepSeek-R1-Distill-Qwen-7B2026.07 | 0.52 | 8,068 | 7,021 | 0.057 | 0.049 | |
| ConCISESimPOBackbone=DeepSeek-R1-Distill-Qwen-7B2026.07 | 0.502 | 6,030 | 5,011 | 0.295 | 0.321 | |
| OverThinkSimPOBackbone=DeepSeek-R1-Distill-Qwen-7B2026.07 | 0.498 | 7,270 | 5,856 | 0.15 | 0.207 | |
| R1-7BBackbone=DeepSeek-R1-Distill-Qwen-7B2026.07 | 0.492 | 8,553 | 7,382 | — | — | |
| CATBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.07 | 0.37 | 9,716 | 8,348 | 0.038 | 0.097 | |
| DASTBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.07 | 0.367 | 9,980 | 8,993 | 0.012 | 0.027 | |
| OverThinkSimPOBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.07 | 0.357 | 8,908 | 7,967 | 0.118 | 0.138 | |
| ConCISESimPOBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.07 | 0.349 | 7,890 | 6,995 | 0.219 | 0.243 | |
| R1-1.5BBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.07 | 0.34 | 10,100 | 9,240 | — | — |