Scientific Reasoning on GPQA Main
67.97AccuracyLLaMA-70B
Evaluation Results
| Method | Links | |
|---|---|---|
| LLaMA-70BSetting=Steer2026.06 | 67.97 | |
| LLaMA-70BSetting=Base2026.06 | 67.19 | |
| Qwen3-32BSetting=Steer2026.06 | 65.96 | |
| Qwen3-32BSetting=Base2026.06 | 64.4 | |
| Qwen3-14BSetting=Steer2026.06 | 60.49 | |
| Qwen3-14BSetting=Base2026.06 | 58.71 | |
| Qwen3-8BSetting=Steer2026.06 | 55.28 | |
| LLaMA-8BSetting=Steer2026.06 | 55.13 | |
| Qwen3-8BSetting=Base2026.06 | 54.69 | |
| Qwen3-4BSetting=Steer2026.06 | 51.33 | |
| LLaMA-8BSetting=Base2026.06 | 50.45 | |
| Qwen3-4BSetting=Base2026.06 | 50.22 | |
| Llama 70B2026.02 | 43.08 | |
| Qwen3-1.7BSetting=Steer2026.06 | 41.07 | |
| Mistral 24B2026.02 | 39.95 | |
| FiMI Instruct2026.02 | 37.95 | |
| Gemma 2 27B2026.02 | 37.72 | |
| Qwen3-1.7BSetting=Base2026.06 | 37.05 | |
| Official Instruct ModelEvaluation Category=Official Instruct Model2026.05 | 35.49 | |
| NeMo 12B IT2026.02 | 34.59 | |
| Mistral 7B2026.02 | 33.92 | |
| Official Instruct ModelProtocol=Official Instruct, Backbone=SmolLM3-3B2026.05 | 33.26 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=2.02026.04 | 30.8 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=2.02026.04 | 29.69 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=10.02026.04 | 29.69 | |
| Opus-4.7 (xHigh)Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 29.32 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=5.02026.04 | 29.24 | |
| Min-pModel=LLaMA3-8B-Instruct, Temperature=1.02026.04 | 29.02 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=3.02026.04 | 29.02 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=1.02026.04 | 29.02 | |
| GPT-5.4 (High)Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 29.02 | |
| Llama 3.1 8B2026.02 | 28.57 | |
| Top-pModel=LLaMA3-8B-Instruct, Temperature=1.02026.04 | 28.57 | |
| Top-kModel=LLaMA3-8B-Instruct, Temperature=1.02026.04 | 28.35 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=10.02026.04 | 28.35 | |
| Opus-4.7 (xHigh)Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 27.83 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=5.02026.04 | 27.46 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=4.02026.04 | 27.46 | |
| GLM-4.7 & ANDESEvaluation Category=Proposed Method Comparison, Base Model=Qwen3-1.7B2026.05 | 27.03 | |
| Top-kModel=LLaMA3-8B-Instruct, Temperature=2.02026.04 | 26.79 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=4.02026.04 | 26.79 | |
| Top-nσModel=LLaMA3-8B-Instruct, Temperature=1.02026.04 | 26.56 | |
| Min-kModel=LLaMA3-8B-Instruct, Temperature=3.02026.04 | 26.56 | |
| Opus-4.6 (1M)Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 26.41 | |
| Min-pModel=LLaMA3-8B-Instruct, Temperature=2.02026.04 | 26.34 | |
| GPT-5.2Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 25.82 | |
| GLM-4.7 & ANDES (Ours)Protocol=Proposed Method, Backbone=SmolLM3-3B2026.05 | 25.8 | |
| Opus-4.6Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 25.52 | |
| GPT-5.1-Codex-MaxEvaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 24.78 | |
| GPT-5.4 (High)Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 24.33 | |
| Opus-4.6 (1M)Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 22.99 | |
| Opus-4.6Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 22.92 | |
| GPT-5.1-Codex-MaxProtocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 21.65 | |
| Top-pModel=LLaMA3-8B-Instruct, Temperature=2.02026.04 | 21.21 | |
| Min-pModel=LLaMA3-8B-Instruct, Temperature=3.02026.04 | 21.21 | |
| GLM-4.7 (Scaffold-only)Evaluation Category=Proposed Method Comparison, Base Model=Qwen3-1.7B2026.05 | 20.13 | |
| GLM-4.7 (Scaffold-only)Protocol=Proposed Method, Backbone=SmolLM3-3B2026.05 | 19.31 | |
| Min-pModel=Qwen3-4B-Instruct, Temperature=1.02026.04 | 19.2 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=4.02026.04 | 19.2 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=1.02026.04 | 18.97 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=2.02026.04 | 18.97 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=10.02026.04 | 18.97 | |
| Top-kModel=Qwen3-4B-Instruct, Temperature=2.02026.04 | 18.75 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=2.02026.04 | 18.75 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=5.02026.04 | 18.53 | |
| Gemini-3.1-ProProtocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 18.01 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=3.02026.04 | 17.86 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=1.02026.04 | 17.41 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=4.02026.04 | 17.41 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=5.02026.04 | 17.41 | |
| Min-kModel=Qwen3-4B-Instruct, Temperature=3.02026.04 | 17.41 | |
| GPT-5.2Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 17.41 | |
| Top-kModel=Qwen3-4B-Instruct, Temperature=1.02026.04 | 17.19 | |
| Top-pModel=Qwen3-4B-Instruct, Temperature=1.02026.04 | 17.19 | |
| MiniMax-M2.5Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 17.19 | |
| Min-pModel=Qwen3-4B-Instruct, Temperature=2.02026.04 | 16.96 | |
| Sonnet-4.5Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 16.74 | |
| Gemini-3.1-ProEvaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 16.15 | |
| Top-pModel=Qwen3-4B-Instruct, Temperature=2.02026.04 | 16.07 | |
| Top-nσModel=Qwen3-4B-Instruct, Temperature=10.02026.04 | 16.07 | |
| Top-kModel=LLaMA3-8B-Instruct, Temperature=3.02026.04 | 15.18 | |
| Min-pModel=Qwen3-4B-Instruct, Temperature=3.02026.04 | 14.73 | |
| MiniMax-M2.1Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 14.51 | |
| Base Model (Qwen3-1.7B)Evaluation Category=Zero-Shot2026.05 | 14.06 | |
| MiniMax-M2.5Evaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 14.06 | |
| Qwen3-MaxEvaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 14.06 | |
| Kimi-K2-ThinkingEvaluation Category=Agent-Post-Trained, Base Model=Qwen3-1.7B2026.05 | 14.06 | |
| GLM-4.7 (OpenCode)Evaluation Category=Proposed Method Comparison, Base Model=Qwen3-1.7B2026.05 | 14.06 | |
| Top-kModel=Qwen3-4B-Instruct, Temperature=3.02026.04 | 12.28 | |
| MiniMax-M2.1Protocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 12.05 | |
| Top-pModel=Qwen3-4B-Instruct, Temperature=3.02026.04 | 10.71 | |
| Min-pModel=Qwen3-4B-Instruct, Temperature=4.02026.04 | 10.04 | |
| Top-kModel=Qwen3-4B-Instruct, Temperature=4.02026.04 | 9.6 | |
| Top-pModel=LLaMA3-8B-Instruct, Temperature=3.02026.04 | 7.59 | |
| Min-pModel=LLaMA3-8B-Instruct, Temperature=4.02026.04 | 6.92 | |
| Top-kModel=LLaMA3-8B-Instruct, Temperature=4.02026.04 | 5.58 | |
| Min-pModel=Qwen3-4B-Instruct, Temperature=5.02026.04 | 5.58 | |
| Top-pModel=Qwen3-4B-Instruct, Temperature=4.02026.04 | 5.36 | |
| Base Model (SmolLM3-3B)Protocol=Zero-Shot, Backbone=SmolLM3-3B2026.05 | 4.91 | |
| Qwen3-MaxProtocol=Agent-Post-Trained, Backbone=SmolLM3-3B2026.05 | 4.91 |