Multiple Choice Question Answering on MMLU-Pro
96.5MMLU-Pro Overall AccuracyBERT-Judge
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BERT-Judgeclean_name=BERT-as-a-Judge2026.04 | 96.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Regex2026.04 | 88.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TopoDIMBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=true, Dec.=true2026.01 | 84.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Sonnet 42025.08 | 83.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | — | — | — | — | |
| G-DesignerBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=true, Dec.=false2026.01 | 82.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ORACLE2026.04 | 82.6 | 85.4 | — | — | 85 | — | 82.8 | — | 85.5 | 80.6 | — | 70.8 | — | — | — | 84.2 | 83.3 | 85.5 | — | |
| AgentDropoutBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=false, Dec.=true2026.01 | 82.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=false, Dec.=false2026.01 | 81.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-DebateBase model=DeepSeek-V3.2-251201:671B, Ada.=false, DIM.=false, Dec.=false2026.01 | 80.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TopoDIMBase model=GPT-OSS-120B, Ada.=true, DIM.=true, Dec.=true2026.01 | 80.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerBase model=GPT-OSS-120B, Ada.=true, DIM.=true, Dec.=false2026.01 | 79.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBase model=DeepSeek-V3.2-251201:671B, Ada.=false, DIM.=false, Dec.=false2026.01 | 78.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaBase model=DeepSeek-V3.2-251201:671B, Ada.=false, DIM.=false, Dec.=false2026.01 | 78.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentDropoutBase model=GPT-OSS-120B, Ada.=true, DIM.=false, Dec.=true2026.01 | 78.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Sonnet 3.5Date=2024-10-222025.08 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2 | — | — | — | — | |
| Gemini 2.0 Flash2025.08 | 77.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | — | — | — | — | |
| GPT-4oDate=2024-11-202025.08 | 77.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4 | — | — | — | — | |
| XGBOOST2026.04 | 77.9 | 80.8 | — | — | 80.5 | — | 79.5 | — | 81.3 | 76 | — | 64.6 | — | — | — | 79.2 | 78.7 | 80.7 | — | |
| TRANSLATE2026.04 | 77.7 | 80.3 | — | — | 80.3 | — | 79.5 | — | 81 | 75.6 | — | 64.5 | — | — | — | 79 | 78.5 | 80.5 | — | |
| MLP2026.04 | 77.7 | 80.8 | — | — | 80.7 | — | 78.2 | — | 81.3 | 75.5 | — | 64.7 | — | — | — | 79.3 | 78 | 79.3 | — | |
| GPTSwarmBase model=GPT-OSS-120B, Ada.=true, DIM.=false, Dec.=false2026.01 | 76.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-DebateBase model=GPT-OSS-120B, Ada.=false, DIM.=false, Dec.=false2026.01 | 75.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaBase model=GPT-OSS-120B, Ada.=false, DIM.=false, Dec.=false2026.01 | 74.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini-2.5-FlashAccess=Proprietary2026.01 | 74.3 | 76.26 | 75.73 | 76.9 | 77.33 | 75.72 | 76.79 | 78.72 | 78.29 | 75.72 | 63.42 | 68.66 | 68.12 | — | — | — | — | — | — | |
| Claude-4.0-SonnetAccess=Proprietary2026.01 | 71.69 | 77.65 | 82.78 | 80.75 | 77.86 | 78.61 | 76.47 | 80 | 81.39 | 70.59 | 35.08 | 55.83 | 63.31 | — | — | — | — | — | — | |
| o4-miniAccess=Proprietary, Size=mini2026.01 | 70.48 | 74.97 | 79.79 | 80.1 | 74.87 | 75.72 | 71.23 | 80.43 | 78.39 | 72.51 | 30.38 | 60.11 | 67.27 | — | — | — | — | — | — | |
| DeepSeek-R1Access=Open-Weight2026.01 | 69.83 | 70.05 | 78.39 | 78.18 | 76.79 | 76.36 | 63.85 | 78.29 | 77.33 | 74.44 | 46.1 | 57.01 | 61.18 | — | — | — | — | — | — | |
| GPT-5Access=Proprietary2026.01 | 69.31 | 75.3 | 76.9 | 77.33 | 74.12 | 74.22 | 68.55 | 79.78 | 79.36 | 70.7 | 36.47 | 55.61 | 63.42 | — | — | — | — | — | — | |
| LLaMA-4-MaverickAccess=Open-Weight2026.01 | 69.3 | 73.9 | 80.85 | 77.65 | 75.83 | 74.33 | 71.98 | 77.33 | 77.22 | 72.09 | 37.32 | 53.48 | 59.68 | — | — | — | — | — | — | |
| NATIVE2026.04 | 69.2 | 80.5 | — | — | 79.7 | — | 35.7 | — | 80.8 | 69.3 | — | 43.4 | — | — | — | 77.9 | 75.4 | 80.1 | — | |
| GPT-4.1Access=Proprietary2026.01 | 68.8 | 74.65 | 77.65 | 77.54 | 75.72 | 74.87 | 71.34 | 79.57 | 79.25 | 74.12 | 31.44 | 46.1 | 63.32 | — | — | — | — | — | — | |
| gpt-oss-120BAccess=Open-Weight, Size=120B2026.01 | 66.88 | 70.27 | 77.22 | 75.93 | 74.12 | 72.94 | 69.84 | 75.83 | 75.83 | 62.03 | 35.4 | 54.44 | 58.72 | — | — | — | — | — | — | |
| DeepSeek-V3Access=Open-Weight2026.01 | 65.4 | 74.98 | 79.68 | 75.72 | 75.72 | 75.4 | 69.2 | 78.82 | 75.3 | 66.2 | 30.16 | 36.9 | 46.74 | — | — | — | — | — | — | |
| GPT-4oAccess=Proprietary2026.01 | 65.37 | 72.73 | 76.26 | 75.72 | 73.58 | 73.26 | 69.52 | 76.37 | 76.37 | 66.41 | 25.99 | 42.68 | 55.51 | — | — | — | — | — | — | |
| GPT-5-miniAccess=Proprietary, Size=mini2026.01 | 63.9 | 73.16 | 76.69 | 74.22 | 73.58 | 71.34 | 64.81 | 74.76 | 76.26 | 61.5 | 19.89 | 44.17 | 56.47 | — | — | — | — | — | — | |
| GPT-4.1-miniAccess=Proprietary, Size=mini2026.01 | 62.99 | 70.05 | 78.07 | 72.41 | 74.97 | 70.91 | 66.1 | 75.51 | 73.05 | 62.03 | 15.51 | 44.06 | 53.16 | — | — | — | — | — | — | |
| Haiku 3.52025.08 | 62.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | 5 | — | — | — | — | |
| LLaMA-4-ScoutAccess=Open-Weight2026.01 | 62.03 | 69.41 | 74.44 | 73.69 | 70.37 | 65.99 | 67.59 | 74.12 | 73.05 | 62.14 | 29.95 | 29.41 | 54.23 | — | — | — | — | — | — | |
| MedGemma-27BAccess=Open-Weight, Size=27B, Domain=Medical2026.01 | 60.62 | 68.88 | 75.29 | 73.8 | 74.01 | 69.2 | 67.27 | 73.15 | 73.69 | 61.28 | 14.86 | 29.73 | 46.31 | — | — | — | — | — | — | |
| Phi-4-ReasoningAccess=Open-Weight, Mode=Reasoning2026.01 | 58.22 | 67.48 | 74.76 | 74.44 | 72.19 | 71.66 | 64.81 | 68.77 | 74.33 | 56.04 | 16.58 | 32.19 | 25.35 | — | — | — | — | — | — | |
| gpt-oss-20BAccess=Open-Weight, Size=20B2026.01 | 57.66 | 63.32 | 64.92 | 69.09 | 67.49 | 64.07 | 63.74 | 68.34 | 69.09 | 49.09 | 22.78 | 42.67 | 47.38 | — | — | — | — | — | — | |
| HuatuoGPT-o1-70BAccess=Open-Weight, Size=70B2026.01 | 57.58 | 60.53 | 72.94 | 69.62 | 69.2 | 61.39 | 65.45 | 70.59 | 70.69 | 57.11 | 24.17 | 30.48 | 38.82 | — | — | — | — | — | — | |
| LLM-Judgebackbone=Qwen-3 0.6B2026.04 | 57.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QwQ-32BAccess=Open-Weight, Size=32B2026.01 | 56.68 | 69.41 | 74.76 | 73.9 | 72.94 | 68.45 | 64.38 | 76.15 | 74.97 | 46.63 | 18.61 | 19.14 | 20.86 | — | — | — | — | — | — | |
| POPModel=Qwen-2.5-7B-Inst, Training Strategy=POP, Evaluation Protocol=0-shot2026.04 | 56.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HuatuoGPT-o1-72BAccess=Open-Weight, Size=72B2026.01 | 56.57 | 69.52 | 75.3 | 73.37 | 68.77 | 70.69 | 65.56 | 74.55 | 73.9 | 44.92 | 20.64 | 24.49 | 17.11 | — | — | — | — | — | — | |
| BaseModel=Qwen-2.5-7B-Inst, Training Strategy=Base, Evaluation Protocol=0-shot2026.04 | 56.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-3.3-70BAccess=Open-Weight, Size=70B2026.01 | 56.2 | 53.15 | 75.94 | 73.58 | 71.98 | 51.87 | 60.32 | 71.66 | 72.83 | 57.65 | 24.06 | 27.17 | 34.23 | — | — | — | — | — | — | |
| TopoDIMBase model=Gemma-3-it:12B, Ada.=true, DIM.=true, Dec.=true2026.01 | 55.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemma-3-27BAccess=Open-Weight, Size=27B2026.01 | 54.78 | 62.46 | 67.38 | 68.98 | 64.49 | 58.5 | 57.97 | 70.27 | 67.7 | 54.54 | 13.15 | 27.59 | 44.28 | — | — | — | — | — | — | |
| Qwen3-14B-thinkingAccess=Open-Weight, Size=14B, Mode=thinking2026.01 | 54.71 | 66.31 | 75.29 | 72.84 | 71.55 | 70.69 | 66.31 | 71.02 | 71.34 | 44.92 | 12.73 | 17.43 | 16.04 | — | — | — | — | — | — | |
| Train on DModel=Qwen-2.5-7B-Inst, Training Strategy=Naive pretraining on D, Evaluation Protocol=0-shot2026.04 | 54.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerBase model=Gemma-3-it:12B, Ada.=true, DIM.=true, Dec.=false2026.01 | 54.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentDropoutBase model=Gemma-3-it:12B, Ada.=true, DIM.=false, Dec.=true2026.01 | 53.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniAccess=Proprietary2026.01 | 53.5 | 57.11 | 71.98 | 68.13 | 65.88 | 60.43 | 56.68 | 66.63 | 68.34 | 48.02 | 16.47 | 27.81 | 34.54 | — | — | — | — | — | — | |
| Qwen2.5-72BAccess=Open-Weight, Size=72B2026.01 | 53.47 | 69.95 | 73.8 | 69.73 | 68.24 | 66.84 | 61.6 | 71.55 | 69.3 | 35.51 | 18.18 | 15.4 | 21.5 | — | — | — | — | — | — | |
| GPTSwarmBase model=Gemma-3-it:12B, Ada.=true, DIM.=false, Dec.=false2026.01 | 53.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-3.1-70BAccess=Open-Weight, Size=70B2026.01 | 53.12 | 59.89 | 72.3 | 66.63 | 67.81 | 59.89 | 52.3 | 66.63 | 68.66 | 50.8 | 20.75 | 22.99 | 28.77 | — | — | — | — | — | — | |
| l-MoEaccActive Params=8.02B, Total Params=72.8B2026.01 | 52.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-DebateBase model=Gemma-3-it:12B, Ada.=false, DIM.=false, Dec.=false2026.01 | 52.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBase model=Gemma-3-it:12B, Ada.=false, DIM.=false, Dec.=false2026.01 | 51.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude-3.5-HaikuAccess=Proprietary2026.01 | 51.82 | 59.36 | 68.45 | 71.66 | 63.1 | 59.89 | 55.61 | 68.45 | 72.62 | 36.36 | 15.51 | 20.96 | 29.84 | — | — | — | — | — | — | |
| l-MoEeffActive Params=5.91B, Total Params=72.8B2026.01 | 51.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaBase model=Gemma-3-it:12B, Ada.=false, DIM.=false, Dec.=false2026.01 | 51.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Long Grounded Thoughtstraining_protocol=SFT Only2025.11 | 50.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LPT2025.11 | 50.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baichuan-M2-32BAccess=Open-Weight, Size=32B2026.01 | 50.56 | 69.52 | 75.4 | 69.41 | 67.91 | 63.85 | 56.04 | 67.91 | 67.59 | 18.61 | 16.36 | 18.29 | 15.83 | — | — | — | — | — | — | |
| GPT-4.1-nanoAccess=Proprietary, Size=nano2026.01 | 50.15 | 56.58 | 72.73 | 65.56 | 65.99 | 54.87 | 47.49 | 66.52 | 68.34 | 37.22 | 14.65 | 21.28 | 30.59 | — | — | — | — | — | — | |
| Qwen3-8B-thinkingAccess=Open-Weight, Size=8B, Mode=thinking2026.01 | 49.79 | 62.46 | 72.3 | 68.87 | 66.52 | 64.92 | 62.57 | 68.13 | 68.13 | 24.06 | 12.94 | 13.58 | 13.05 | — | — | — | — | — | — | |
| Qwen3-14BAccess=Open-Weight, Size=14B2026.01 | 48.54 | 65.13 | 71.34 | 65.24 | 65.99 | 58.4 | 57.22 | 68.77 | 65.67 | 24.38 | 11.34 | 12.83 | 16.15 | — | — | — | — | — | — | |
| Phi-4Access=Open-Weight2026.01 | 48.46 | 51.66 | 72.51 | 65.56 | 68.02 | 56.68 | 48.88 | 68.02 | 66.31 | 33.58 | 13.58 | 18.61 | 18.08 | — | — | — | — | — | — | |
| BaselineActive Params=8.09B, Total Params=72.6B2026.01 | 48.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-VL-7B-Instruct2025.11 | 47.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Long Grounded Thoughtstraining_protocol=SFT + DPO2025.11 | 47.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemma-3-12BAccess=Open-Weight, Size=12B2026.01 | 47.02 | 51.12 | 61.5 | 58.71 | 58.39 | 53.05 | 51.87 | 59.04 | 58.61 | 45.99 | 9.41 | 23.63 | 32.94 | — | — | — | — | — | — | |
| Long Grounded Thoughtstraining_protocol=SFT + GRPO2025.11 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral-Small-3.1-24BAccess=Open-Weight, Size=24B, Version=3.12026.01 | 46.04 | 56.79 | 70.05 | 65.24 | 64.92 | 58.93 | 48.98 | 65.13 | 65.78 | 23.64 | 11.34 | 10.48 | 11.23 | — | — | — | — | — | — | |
| GPT-5-nanoAccess=Proprietary, Size=nano2026.01 | 45.05 | 47.59 | 65.78 | 58.29 | 60.32 | 52.83 | 40.43 | 56.47 | 59.68 | 39.14 | 12.51 | 20.75 | 26.84 | — | — | — | — | — | — | |
| DeepSeek-R1-Qwen3-8BAccess=Open-Weight, Backbone=Qwen3, Size=8B2026.01 | 44.76 | 61.39 | 68.23 | 58.4 | 61.93 | 54.87 | 56.9 | 63.74 | 59.68 | 19.57 | 10.91 | 10.91 | 10.59 | — | — | — | — | — | — | |
| Qwen2.5-14BAccess=Open-Weight, Size=14B2026.01 | 44.73 | 59.57 | 68.13 | 57.97 | 57.75 | 52.51 | 49.2 | 60.86 | 59.89 | 20.43 | 16.9 | 14.12 | 19.47 | — | — | — | — | — | — | |
| Qwen3-4B-thinkingAccess=Open-Weight, Size=4B, Mode=thinking2026.01 | 44.27 | 58.18 | 63.75 | 64.39 | 60.75 | 57.43 | 53.58 | 61.5 | 61.6 | 15.3 | 12.08 | 12.84 | 9.84 | — | — | — | — | — | — | |
| Haiku 32025.08 | 42.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | 6 | — | — | — | — | |
| Qwen3-8BAccess=Open-Weight, Size=8B2026.01 | 41.97 | 58.5 | 66.95 | 54.65 | 54.76 | 52.3 | 45.67 | 56.47 | 57.86 | 19.68 | 11.66 | 13.05 | 12.09 | — | — | — | — | — | — | |
| Train on DModel=Qwen-2.5-7B, Training Strategy=Naive pretraining on D, Evaluation Protocol=0-shot2026.04 | 41.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HuatuoGPT-o1-7BAccess=Open-Weight, Size=7B2026.01 | 39.38 | 53.8 | 58.29 | 57.43 | 53.16 | 48.77 | 41.39 | 59.36 | 58.61 | 10.37 | 11.34 | 9.52 | 10.48 | — | — | — | — | — | — | |
| Virgo2025.11 | 37.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MonoSoupSetting=M-1 (Linear)2026.02 | 37.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MonoSoup R = 0.8Setting=M-1 (Linear)2026.02 | 37.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MonoSoupSetting=M-2 (Cosine)2026.02 | 36.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| POPModel=Qwen-2.5-7B, Training Strategy=POP, Evaluation Protocol=0-shot2026.04 | 36.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModelStock (M-1, M-2)Setting=Pairwise2026.02 | 36.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LINESSetting=M-1 (Linear)2026.02 | 36.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModelStock (M-2, M-3)Setting=Pairwise2026.02 | 36.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-4BAccess=Open-Weight, Size=4B2026.01 | 36.6 | 53.16 | 59.15 | 50.8 | 50.59 | 43.64 | 37.75 | 48.34 | 50.27 | 12.51 | 11.23 | 11.02 | 10.7 | — | — | — | — | — | — | |
| MonoSoup R = 0.8Setting=M-2 (Cosine)2026.02 | 36.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OneVision-8BModel=OneVision-8B2026.06 | 36.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModelStock (M-1, M-3)Setting=Pairwise2026.02 | 36.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HuatuoGPT-o1-8BAccess=Open-Weight, Size=8B2026.01 | 36.32 | 47.06 | 54.33 | 51.66 | 49.52 | 41.18 | 35.61 | 48.66 | 45.24 | 28.88 | 11.66 | 11.76 | 10.27 | — | — | — | — | — | — | |
| LINESSetting=M-2 (Cosine)2026.02 | 36.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StandardSetting=M-2 (Cosine)2026.02 | 35.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |