List your company
ResearchFormerly WizWand

Graduate-level Reasoning on GPQA Diamond (Accuracy)

98.74AccuracyHEART
11.858434.414256.9779.5258May 21, 2025Jul 21, 2025Sep 20, 2025Nov 21, 2025Jan 21, 2026Mar 23, 2026May 24, 2026
Updated 5mo ago

Evaluation results

MethodLinks
HEARTModel=Claude 4 Sonnet2025.0998.74
HEARTModel=Gemini 2.5 Flash2025.0990.06
HEARTModel=Deepseek-Reasoner2025.0988.05
HEARTModel=Gemma3 12b Instruct2025.0985.41
VanillaModel=Gemini 2.5 Flash2025.0974.47
CDGBase Model=DeepSeek-R1-8B2026.0572.2
DeepConf-TailBase Model=DeepSeek-R1-8B, Filtering=top 10%2026.0570.7
HEARTModel=Gemma3 4b Instruct2025.0970.19
DeepConf-MeanBase Model=DeepSeek-R1-8B, Filtering=top 10%2026.0568.7
Majority Vote (Self-Consistency)Base Model=DeepSeek-R1-8B2026.0568.2
VanillaModel=Deepseek-Reasoner2025.0961.48
SPIRALBackbone=DeepSeek-Distill-Qwen-7B, Training Data=Multi-game2025.0649.6
DeepSeek-Distill-Qwen-7BBackbone=DeepSeek-Distill-Qwen-7B2025.0648.6
Qwen3-14BPrecision=BF162026.0244.95
SPIRALBackbone=Qwen3-8B, Training Data=Multi-game2025.0644.6
SFTBackbone=DeepSeek-Distill-Qwen-7B, Training Data=Multi-game2025.0644.5
Qwen3-14BPrecision=FP4 w. Attn-QAT2026.0243.94
SFTBackbone=Qwen3-8B, Training Data=Multi-game2025.0641.6
Llama3.1-70BPrecision=BF162026.0240.91
SPIRALBackbone=Qwen3-4B, Training Data=Multi-game2025.0640.1
Llama3.1-70BPrecision=FP4 w. Attn-QAT2026.0238.38
Qwen3-8B-BaseBackbone=Qwen3-8B, Evaluation Protocol=Few-shot2025.0638
TemplateRLBase Model=Qwen2.5-Math-7B-Base2025.0537.9
SFTBackbone=Qwen3-4B, Training Data=Multi-game2025.0637.8
SPIRALBackbone=Qwen3-4B, Training Data=Kuhn Poker2025.0637
SPIRALBackbone=Octothinker-8B, Training Data=Multi-game2025.0633.8
SFTBackbone=Qwen3-4B, Training Data=Kuhn Poker2025.0633
SPIRALBackbone=Llama-3.1-8B-Instruct, Training Data=Multi-game2025.0632.2
GRPOBase Model=Qwen2.5-Math-7B-Base2025.0531.3
Qwen3-4B-BaseBackbone=Qwen3-4B, Evaluation Protocol=Few-shot2025.0630.6
OpenReasoner-ZeroBase Model=Qwen2.5-Math-7B-Base2025.0530.3
Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct2025.0630.2
SFTBackbone=Llama-3.1-8B-Instruct, Training Data=Multi-game2025.0630
Qwen2.5-Math-7B-Instruct2025.0528.8
Oat-ZeroBase Model=Qwen2.5-Math-7B-Base2025.0525.8
SFTBackbone=Octothinker-8B, Training Data=Multi-game2025.0624.9
Octothinker-8B-BaseBackbone=Octothinker-8B2025.0622.1
SimpleRL-ZeroBase Model=Qwen2.5-Math-7B-Base2025.0521.2
PRIME-ZeroBase Model=Qwen2.5-Math-7B-Base2025.0520.2
Qwen2.5-Math-7B-Base2025.0515.2