Graduate-level Reasoning on GPQA Diamond (Accuracy)
98.74AccuracyHEART
Evaluation Results
| Method | Links | |
|---|---|---|
| HEARTModel=Claude 4 Sonnet2025.09 | 98.74 | |
| HEARTModel=Gemini 2.5 Flash2025.09 | 90.06 | |
| HEARTModel=Deepseek-Reasoner2025.09 | 88.05 | |
| HEARTModel=Gemma3 12b Instruct2025.09 | 85.41 | |
| VanillaModel=Gemini 2.5 Flash2025.09 | 74.47 | |
| CDGBase Model=DeepSeek-R1-8B2026.05 | 72.2 | |
| DeepConf-TailBase Model=DeepSeek-R1-8B, Filtering=top 10%2026.05 | 70.7 | |
| HEARTModel=Gemma3 4b Instruct2025.09 | 70.19 | |
| DeepConf-MeanBase Model=DeepSeek-R1-8B, Filtering=top 10%2026.05 | 68.7 | |
| Majority Vote (Self-Consistency)Base Model=DeepSeek-R1-8B2026.05 | 68.2 | |
| VanillaModel=Deepseek-Reasoner2025.09 | 61.48 | |
| SPIRALBackbone=DeepSeek-Distill-Qwen-7B, Training Data=Multi-game2025.06 | 49.6 | |
| DeepSeek-Distill-Qwen-7BBackbone=DeepSeek-Distill-Qwen-7B2025.06 | 48.6 | |
| Qwen3-14BPrecision=BF162026.02 | 44.95 | |
| SPIRALBackbone=Qwen3-8B, Training Data=Multi-game2025.06 | 44.6 | |
| SFTBackbone=DeepSeek-Distill-Qwen-7B, Training Data=Multi-game2025.06 | 44.5 | |
| Qwen3-14BPrecision=FP4 w. Attn-QAT2026.02 | 43.94 | |
| SFTBackbone=Qwen3-8B, Training Data=Multi-game2025.06 | 41.6 | |
| Llama3.1-70BPrecision=BF162026.02 | 40.91 | |
| SPIRALBackbone=Qwen3-4B, Training Data=Multi-game2025.06 | 40.1 | |
| Llama3.1-70BPrecision=FP4 w. Attn-QAT2026.02 | 38.38 | |
| Qwen3-8B-BaseBackbone=Qwen3-8B, Evaluation Protocol=Few-shot2025.06 | 38 | |
| TemplateRLBase Model=Qwen2.5-Math-7B-Base2025.05 | 37.9 | |
| SFTBackbone=Qwen3-4B, Training Data=Multi-game2025.06 | 37.8 | |
| SPIRALBackbone=Qwen3-4B, Training Data=Kuhn Poker2025.06 | 37 | |
| SPIRALBackbone=Octothinker-8B, Training Data=Multi-game2025.06 | 33.8 | |
| SFTBackbone=Qwen3-4B, Training Data=Kuhn Poker2025.06 | 33 | |
| SPIRALBackbone=Llama-3.1-8B-Instruct, Training Data=Multi-game2025.06 | 32.2 | |
| GRPOBase Model=Qwen2.5-Math-7B-Base2025.05 | 31.3 | |
| Qwen3-4B-BaseBackbone=Qwen3-4B, Evaluation Protocol=Few-shot2025.06 | 30.6 | |
| OpenReasoner-ZeroBase Model=Qwen2.5-Math-7B-Base2025.05 | 30.3 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct2025.06 | 30.2 | |
| SFTBackbone=Llama-3.1-8B-Instruct, Training Data=Multi-game2025.06 | 30 | |
| Qwen2.5-Math-7B-Instruct2025.05 | 28.8 | |
| Oat-ZeroBase Model=Qwen2.5-Math-7B-Base2025.05 | 25.8 | |
| SFTBackbone=Octothinker-8B, Training Data=Multi-game2025.06 | 24.9 | |
| Octothinker-8B-BaseBackbone=Octothinker-8B2025.06 | 22.1 | |
| SimpleRL-ZeroBase Model=Qwen2.5-Math-7B-Base2025.05 | 21.2 | |
| PRIME-ZeroBase Model=Qwen2.5-Math-7B-Base2025.05 | 20.2 | |
| Qwen2.5-Math-7B-Base2025.05 | 15.2 |