Scientific Reasoning on GPQA Diamond (test)
99.37AccuracyHEART
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HEARTModel=Gemini 3 Pro, Prompt Strategy=HEART, Setting=Human-in-the-Loop Proxy2025.09 | 99.37 | — | |
| WaitModel=Gemini 3 Pro, Prompt Strategy=Wait, Setting=Human-in-the-Loop Proxy2025.09 | 95.35 | — | |
| Self ReflectionModel=Gemini 3 Pro, Prompt Strategy=Self Reflection, Setting=Human-in-the-Loop Proxy2025.09 | 93.84 | — | |
| CoTModel=Gemini 3 Pro, Prompt Strategy=CoT, Setting=Human-in-the-Loop Proxy2025.09 | 92.83 | — | |
| VanillaModel=Gemini 3 Pro, Prompt Strategy=Vanilla, Setting=Human-in-the-Loop Proxy2025.09 | 91.7 | — | |
| HEARTModel=Gemini 3 Flash, Prompt Strategy=HEART, Setting=Human-in-the-Loop Proxy2025.09 | 90.19 | — | |
| T3-StructModel=GPT-52026.05 | 87.4 | 4.3 | |
| T3-SemanticModel=GPT-52026.05 | 86.4 | 3.1 | |
| Wikipedia-RPJModel=GPT-52026.05 | 85.9 | 2.5 | |
| Self ReflectionModel=Gemini 3 Flash, Prompt Strategy=Self Reflection, Setting=Human-in-the-Loop Proxy2025.09 | 85.03 | — | |
| GitHubModel=GPT-52026.05 | 84.8 | 1.2 | |
| ArxivModel=GPT-52026.05 | 84.8 | 1.2 | |
| Tavily Search APIModel=GPT-52026.05 | 84.8 | 1.2 | |
| Chunked traj.Model=GPT-52026.05 | 84.8 | 1.2 | |
| WaitModel=Gemini 3 Flash, Prompt Strategy=Wait, Setting=Human-in-the-Loop Proxy2025.09 | 84.78 | — | |
| Wikipedia-DPRModel=GPT-52026.05 | 84.3 | 0.6 | |
| T3-ReflectModel=GPT-52026.05 | 84.3 | 0.6 | |
| No RAGModel=GPT-52026.05 | 83.8 | — | |
| CoTModel=Gemini 3 Flash, Prompt Strategy=CoT, Setting=Human-in-the-Loop Proxy2025.09 | 83.77 | — | |
| StackExchangeModel=GPT-52026.05 | 83.3 | -0.6 | |
| OpenWebMathModel=GPT-52026.05 | 82.8 | -1.2 | |
| CompactDSModel=GPT-52026.05 | 82.8 | -1.2 | |
| GitHubModel=Gemini-2.5 Flash2026.05 | 80.8 | 4.5 | |
| Full traj.Model=GPT-52026.05 | 80.8 | -3.6 | |
| T3-StructModel=Gemini-2.5 Flash2026.05 | 80.8 | 4.5 | |
| Wikipedia-DPRModel=Gemini-2.5 Flash2026.05 | 80.3 | 3.9 | |
| Tavily Search APIModel=Gemini-2.5 Flash2026.05 | 79.8 | 3.2 | |
| Chunked traj.Model=Gemini-2.5 Flash2026.05 | 79.3 | 2.6 | |
| T3-ReflectModel=Gemini-2.5 Flash2026.05 | 79.3 | 2.6 | |
| T3-SemanticModel=Gemini-2.5 Flash2026.05 | 78.8 | 1.9 | |
| Wikipedia-RPJModel=Gemini-2.5 Flash2026.05 | 78.3 | 1.3 | |
| No RAGModel=Gemini-2.5 Flash2026.05 | 77.3 | — | |
| ArxivModel=Gemini-2.5 Flash2026.05 | 77.3 | 0 | |
| CompactDSModel=Gemini-2.5 Flash2026.05 | 77.3 | 0 | |
| StackExchangeModel=Gemini-2.5 Flash2026.05 | 76.8 | -0.6 | |
| Full traj.Model=Gemini-2.5 Flash2026.05 | 76.3 | -1.3 | |
| OpenWebMathModel=Gemini-2.5 Flash2026.05 | 75.8 | -1.9 | |
| T3-SemanticModel=GPT-OSS 120B2026.05 | 74.7 | 5.7 | |
| Wikipedia-RPJModel=GPT-OSS 120B2026.05 | 72.2 | 2.1 | |
| VanillaModel=Gemini 3 Flash, Prompt Strategy=Vanilla, Setting=Human-in-the-Loop Proxy2025.09 | 71.82 | — | |
| Wikipedia-DPRModel=GPT-OSS 120B2026.05 | 71.7 | 1.4 | |
| Chunked traj.Model=GPT-OSS 120B2026.05 | 71.7 | 1.4 | |
| T3-ReflectModel=GPT-OSS 120B2026.05 | 71.7 | 1.4 | |
| No RAGModel=GPT-OSS 120B2026.05 | 70.7 | — | |
| T3-StructModel=GPT-OSS 120B2026.05 | 70.7 | 0 | |
| ArxivModel=GPT-OSS 120B2026.05 | 69.7 | -1.4 | |
| OpenWebMathModel=GPT-OSS 120B2026.05 | 69.2 | -2.1 | |
| Full traj.Model=GPT-OSS 120B2026.05 | 69.2 | -2.1 | |
| GitHubModel=GPT-OSS 120B2026.05 | 68.2 | -3.5 | |
| CompactDSModel=GPT-OSS 120B2026.05 | 67.7 | -4.2 | |
| SSR-DBackbone=Qwen3-32B-Think2026.02 | 65.2 | — | |
| RCSDBackbone=Qwen3-8B, Training Steps=1002026.06 | 64.5 | — | |
| w/o TrainBackbone=Qwen3-32B-Think2026.02 | 64.1 | — | |
| OPSDBackbone=Qwen3-8B, Training Steps=1002026.06 | 63.6 | — | |
| GRPOBackbone=Qwen3-8B, Training Steps=5002026.06 | 63.5 | — | |
| SSRBackbone=Qwen3-32B-Think2026.02 | 62.7 | — | |
| Rubric-GRPOBackbone=Qwen3-8B2026.06 | 62.1 | — | |
| SFTBackbone=Qwen3-8B2026.06 | 62 | — | |
| AUG-SUPBackbone=Qwen3-32B-Think2026.02 | 61.1 | — | |
| NEUBackbone=Qwen3-32B-Think2026.02 | 60.6 | — | |
| Qwen3-8BBackbone=Qwen3-8B2026.06 | 60.6 | — | |
| SUPBackbone=Qwen3-32B-Think2026.02 | 60.1 | — | |
| Tavily Search APIModel=GPT-OSS 120B2026.05 | 59.6 | -15.7 | |
| w/o TrainBackbone=Qwen3-8B-Think2026.02 | 58.1 | — | |
| SSR-DBackbone=Qwen3-8B-Think2026.02 | 56.6 | — | |
| SSRBackbone=Qwen3-8B-Think2026.02 | 53.2 | — | |
| NEMOTRON-NANO-12B-V2 (Base + Post)Pre-training objective=Next-token prediction, Post-training pipeline=SFT + RLVR, Evaluation protocol=Pass@1 (4 runs)2025.09 | 52.15 | — | |
| AUG-SUPBackbone=Qwen3-8B-Think2026.02 | 51.2 | — | |
| SUPBackbone=Qwen3-8B-Think2026.02 | 51 | — | |
| NEMOTRON-NANO-12B-V2 (RLP + Post)Pre-training objective=RLP, Post-training pipeline=SFT + RLVR, Evaluation protocol=Pass@1 (4 runs)2025.09 | 49.62 | — | |
| NEUBackbone=Qwen3-8B-Think2026.02 | 49.5 | — | |
| NEMOTRON-NANO-12B-V2 (RLP)Pre-training objective=RLP, Post-training pipeline=None, Evaluation protocol=Pass@1 (4 runs)2025.09 | 48.86 | — | |
| NEMOTRON-NANO-12B-V2 (RLP + Post)Pre-training objective=RLP, Post-training pipeline=SFT + RLVR, Evaluation protocol=Greedy2025.09 | 48 | — | |
| NEMOTRON-NANO-12B-V2 (Base + Post)Pre-training objective=Next-token prediction, Post-training pipeline=SFT + RLVR, Evaluation protocol=Greedy2025.09 | 41.41 | — | |
| NEMOTRON-NANO-12B-V2 (RLP)Pre-training objective=RLP, Post-training pipeline=None, Evaluation protocol=Greedy2025.09 | 39.9 | — | |
| IntroLLMBackbone=Qwen3-4B-Base2026.02 | 38.26 | — | |
| EADBackbone=Qwen3-4B-Base, annealing range=1.2→0.12026.02 | 38.19 | — | |
| TampoBackbone=Qwen3-4B-Base, temperature range=0.6-1.52026.02 | 36.05 | — | |
| GRPOBackbone=Qwen3-4B-Base, temperature (τ)=1.02026.02 | 35.2 | — | |
| StackExchangeModel=GPT-OSS 120B2026.05 | 34.3 | -51.5 | |
| IntroLLMBackbone=Qwen3-1.7B-Base2026.02 | 30.62 | — | |
| EADBackbone=Qwen3-1.7B-Base, annealing range=1.2→0.12026.02 | 29.36 | — | |
| GRPOBackbone=Qwen3-1.7B-Base, temperature (τ)=1.02026.02 | 28.48 | — | |
| TampoBackbone=Qwen3-1.7B-Base, temperature range=0.6-1.52026.02 | 27.08 | — | |
| BaseBackbone=Qwen3-4B-Base2026.02 | 26.77 | — | |
| NEMOTRON-NANO-12B-V2 (Base)Pre-training objective=Next-token prediction, Post-training pipeline=None, Evaluation protocol=Greedy2025.09 | 25.25 | — | |
| NEMOTRON-NANO-12B-V2 (Base)Pre-training objective=Next-token prediction, Post-training pipeline=None, Evaluation protocol=Pass@1 (4 runs)2025.09 | 22.47 | — | |
| BaseBackbone=Qwen3-1.7B-Base2026.02 | 21.72 | — |