Multi-task Language Understanding on MMLU (test)
90.46Normalized AccuracyOracle
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oracle2024.08 | 90.46 | 1.75 | — | — | — | — | — | — | — | — | — | |
| Full ReasoningBase Model=DeepSeek-R1-Qwen2.5-14b2026.01 | 81.6 | 105.67 | 1,465 | — | — | — | — | — | — | — | — | |
| SyncThinkBase Model=DeepSeek-R1-Qwen2.5-14b2026.01 | 77.99 | 61.59 | 852 | — | — | — | — | — | — | — | — | |
| MASPRM-7BSearch=MCTS(10,3), Agent=Qwen2.5-1.5B-Instruct2025.10 | 75.2 | — | — | 0.2 | 12.4 | — | — | — | — | — | — | |
| MASPRM-7BSearch=SBS(5,3), Agent=Qwen2.5-1.5B-Instruct2025.10 | 74.8 | — | — | 0.7 | 35 | — | — | — | — | — | — | |
| SFT-DPO + MergedBase Model=Qwen2.5-7B-Instruct, Alignment=SFT-DPO, Strategy=Merge2026.02 | 72.93 | — | — | — | — | — | — | — | — | — | — | |
| MASPRM-7BSearch=SBS(5,1), Agent=Qwen2.5-1.5B-Instruct2025.10 | 72.4 | — | — | 0.2 | 15 | — | — | — | — | — | — | |
| SFT + OGPSABase Model=Llama3.1-8B-Instruct, Alignment=SFT, Strategy=OGPSA2026.02 | 70.93 | — | — | — | — | — | — | — | — | — | — | |
| Gemma2-9B (Baseline)MP=false, bits=16, Backbone=Gemma2-9B2026.02 | 70.5 | — | — | — | — | — | — | — | — | — | — | |
| SFT-DPO + MergedBase Model=Llama3.1-8B-Instruct, Alignment=SFT-DPO, Strategy=Merge2026.02 | 70.21 | — | — | — | — | — | — | — | — | — | — | |
| MASPRM-1.5BSearch=MCTS(10,3), Agent=Qwen2.5-1.5B-Instruct2025.10 | 68.8 | — | — | 0.2 | 12.4 | — | — | — | — | — | — | |
| FP16Model=Llama-3-8B, Quantization=W3A162026.01 | 67.31 | — | — | — | — | — | — | — | — | — | — | |
| MASPRM-7BSearch=SC@5, Agent=Qwen2.5-1.5B-Instruct2025.10 | 67.2 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| ScaleBITS + RTNMP=true, bits=3.1, Backbone=Gemma2-9B2026.02 | 67.12 | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=Llama-3-8B, Quantization Precision=3-bit2025.12 | 66.5 | — | — | — | — | — | — | — | — | — | — | |
| ORM-1.5BSearch=MCTS(10,3), Agent=Qwen2.5-1.5B-Instruct2025.10 | 66.2 | — | — | 0.2 | 12.1 | — | — | — | — | — | — | |
| SELECTLLMPolicy=MLC + WEIGHTEDMAXCONF, s (number of LLMs)=22024.08 | 65.81 | 4.78 | — | — | — | — | — | — | — | — | — | |
| Top-s LLMss (number of LLMs)=22024.08 | 65.75 | 16.4 | — | — | — | — | — | — | — | — | — | |
| SELECTLLMPolicy=MLC + LABELLEDMAXCONF, s (number of LLMs)=22024.08 | 65.68 | 4.78 | — | — | — | — | — | — | — | — | — | |
| SELECTLLMPolicy=MLC + MAXCONF, s (number of LLMs)=22024.08 | 65.68 | 4.78 | — | — | — | — | — | — | — | — | — | |
| RTN-g128MP=false, bits=3.1, Backbone=Gemma2-9B2026.02 | 65.52 | — | — | — | — | — | — | — | — | — | — | |
| Llama 3-8BShots=5-shot2024.12 | 65.2 | — | — | — | — | — | — | — | — | — | — | |
| Llama 3-8B E8T2Shots=0-shot2024.12 | 64.1 | — | — | — | — | — | — | — | — | — | — | |
| Llama 3-8B E8T2Shots=5-shot2024.12 | 64 | — | — | — | — | — | — | — | — | — | — | |
| HeRo-QModel=Llama-3-8B, Quantization=W3A162026.01 | 63.8 | — | — | — | — | — | — | — | — | — | — | |
| SELECTLLMPolicy=MLC + WEIGHTEDMAXCONF, s (number of LLMs)=12024.08 | 63.52 | 2.97 | — | — | — | — | — | — | — | — | — | |
| FP16Model=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 63.4 | — | — | — | — | — | — | — | — | — | — | |
| Full ReasoningBase Model=DeepSeek-R1-LLaMA3.1-8b2026.01 | 62.8 | 99.38 | 2,076 | — | — | — | — | — | — | — | — | |
| Unilogit+KLBackbone=Llama 3.1 8B, variant=22025.05 | 62.6 | — | — | — | — | — | — | — | — | — | — | |
| Llama 3-8BShots=0-shot2024.12 | 62.1 | — | — | — | — | — | — | — | — | — | — | |
| Full ReasoningBase Model=DeepSeek-R1-Qwen2.5-7b2026.01 | 61.15 | 73 | 1,676 | — | — | — | — | — | — | — | — | |
| All LLMs2024.08 | 60.92 | 16.4 | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=Llama 3.1 8B2025.05 | 60.8 | — | — | — | — | — | — | — | — | — | — | |
| ME+GDBackbone=Llama 3.1 8B2025.05 | 60.8 | — | — | — | — | — | — | — | — | — | — | |
| SpinQuantModel=Llama-3-8B, Quantization=W3A162026.01 | 60.5 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=MCTS(10,3), Scorer=Policy Likelihood2025.10 | 60.5 | — | — | 0.2 | 12.4 | — | — | — | — | — | — | |
| MASPRM-1.5BSearch=SBS(5,3), Agent=Qwen2.5-1.5B-Instruct2025.10 | 60.4 | — | — | 0.8 | 35 | — | — | — | — | — | — | |
| LLM-Blender2024.08 | 60.27 | 16.4 | — | — | — | — | — | — | — | — | — | |
| CALMModel=Llama-3-8B, Quantization Precision=3-bit2025.12 | 60.2 | — | — | — | — | — | — | — | — | — | — | |
| SyncThinkBase Model=DeepSeek-R1-LLaMA3.1-8b2026.01 | 59.98 | 41.91 | 872 | — | — | — | — | — | — | — | — | |
| OmniQuantModel=Llama-3-8B, Quantization=W3A162026.01 | 59.8 | — | — | — | — | — | — | — | — | — | — | |
| NPO+KLBackbone=Llama 3.1 8B2025.05 | 59.6 | — | — | — | — | — | — | — | — | — | — | |
| Undial+KLBackbone=Llama 3.1 8B, variant=12025.05 | 59.6 | — | — | — | — | — | — | — | — | — | — | |
| Unilogit+KLBackbone=Llama 3.1 8B, variant=12025.05 | 59.6 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=SBS(5,3), Scorer=Policy Likelihood2025.10 | 59.4 | — | — | 0.6 | 35 | — | — | — | — | — | — | |
| Undial+KLBackbone=Llama 3.1 8B, variant=22025.05 | 59.1 | — | — | — | — | — | — | — | — | — | — | |
| SyncThinkBase Model=DeepSeek-R1-Qwen2.5-7b2026.01 | 59.1 | 28.13 | 643 | — | — | — | — | — | — | — | — | |
| MASPRM-1.5BSearch=SBS(5,1), Agent=Qwen2.5-1.5B-Instruct2025.10 | 58.7 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| Single BestBase model=Qwen2.5-1.5B2026.05 | 58.6 | — | — | — | — | — | — | — | — | — | — | |
| SpinQuantModel=Llama-3-8B, Quantization Precision=3-bit2025.12 | 58.5 | — | — | — | — | — | — | — | — | — | — | |
| ORM-1.5BSearch=SC@5, Agent=Qwen2.5-1.5B-Instruct2025.10 | 58.5 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| MASPRM-1.5BSearch=SC@5, Agent=Qwen2.5-1.5B-Instruct2025.10 | 58.3 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| Random2024.08 | 58.2 | 8.27 | — | — | — | — | — | — | — | — | — | |
| ScaleBITS + RTNMP=true, bits=2.1, Backbone=Gemma2-9B2026.02 | 57.89 | — | — | — | — | — | — | — | — | — | — | |
| DAREBase model=Qwen2.5-1.5B2026.05 | 57.6 | — | — | — | — | — | — | — | — | — | — | |
| EvoGMBase model=Qwen2.5-1.5B2026.05 | 57.6 | — | — | — | — | — | — | — | — | — | — | |
| CALMModel=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 57.5 | — | — | — | — | — | — | — | — | — | — | |
| Model SoupBase model=Qwen2.5-1.5B2026.05 | 57.4 | — | — | — | — | — | — | — | — | — | — | |
| GA+KLBackbone=Llama 3.1 8B2025.05 | 57.3 | — | — | — | — | — | — | — | — | — | — | |
| Model SwarmBase model=Qwen2.5-1.5B2026.05 | 57.3 | — | — | — | — | — | — | — | — | — | — | |
| SmoothQuantModel=Llama-3-8B, Quantization Precision=3-bit2025.12 | 57.2 | — | — | — | — | — | — | — | — | — | — | |
| Task ArithmeticBase model=Qwen2.5-1.5B2026.05 | 57 | — | — | — | — | — | — | — | — | — | — | |
| CMABase model=Qwen2.5-1.5B2026.05 | 56.9 | — | — | — | — | — | — | — | — | — | — | |
| MTLBase model=Qwen2.5-1.5B2026.05 | 56.6 | — | — | — | — | — | — | — | — | — | — | |
| BaseBase model=Qwen2.5-1.5B2026.05 | 56.4 | — | — | — | — | — | — | — | — | — | — | |
| TIESBase model=Qwen2.5-1.5B2026.05 | 55.7 | — | — | — | — | — | — | — | — | — | — | |
| SpinQuantModel=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 55 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=SBS(5,1), Scorer=Policy Likelihood2025.10 | 55 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=SC@5, Scorer=Policy Likelihood2025.10 | 54.5 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| AWQModel=Llama-3-8B, Quantization Precision=3-bit2025.12 | 54.2 | — | — | — | — | — | — | — | — | — | — | |
| PSO-MergingBase model=Qwen2.5-1.5B2026.05 | 53.9 | — | — | — | — | — | — | — | — | — | — | |
| SmoothQuantModel=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 53.8 | — | — | — | — | — | — | — | — | — | — | |
| No ReasoningBase Model=DeepSeek-R1-Qwen2.5-7b2026.01 | 53.26 | 12.01 | 276 | — | — | — | — | — | — | — | — | |
| QuantGuardModel=Phi-2-2.7B, Attack=Non-adaptive, Defense Status=After QuantGuard, Quantization=INT82026.06 | 53.2 | — | — | — | — | — | — | — | — | — | — | |
| Non-adaptive attackModel=Phi-2-2.7B, Attack=Non-adaptive, Defense Status=Before Defense, Quantization=INT82026.06 | 52.9 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=SC@5, Scorer=None2025.10 | 52.7 | — | — | 0.3 | 15 | — | — | — | — | — | — | |
| QuantGuardModel=Phi-2-2.7B, Attack=Adaptive, Defense Status=After QuantGuard, Quantization=INT82026.06 | 52.3 | — | — | — | — | — | — | — | — | — | — | |
| Adaptive attackModel=Phi-2-2.7B, Attack=Adaptive, Defense Status=Before Defense, Quantization=INT82026.06 | 51.3 | — | — | — | — | — | — | — | — | — | — | |
| AWQModel=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 50.5 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5B-InstructSearch=Single-Pass, Scorer=None2025.10 | 48.4 | — | — | 0.1 | 3 | — | — | — | — | — | — | |
| FC-MoEalpha=1.0, distribution=non-IID2026.06 | 46.95 | — | — | — | — | — | — | — | — | 46.5 | 46.94 | |
| FP16Model=Llama-3.2-1B, Quantization=W3A162026.01 | 46.38 | — | — | — | — | — | — | — | — | — | — | |
| GPTQModel=Llama-3-8B, Quantization Precision=3-bit2025.12 | 45 | — | — | — | — | — | — | — | — | — | — | |
| GPTQModel=Llama-3.2-3B, Quantization Precision=3-bit2025.12 | 45 | — | — | — | — | — | — | — | — | — | — | |
| FedMoEalpha=1.0, distribution=non-IID2026.06 | 44.63 | — | — | — | — | — | — | — | — | 44.12 | 44.6 | |
| PFL-MoEalpha=1.0, distribution=non-IID2026.06 | 43.05 | — | — | — | — | — | — | — | — | 42.42 | 42.75 | |
| HeRo-QModel=Llama-3.2-1B, Quantization=W3A162026.01 | 42.5 | — | — | — | — | — | — | — | — | — | — | |
| FC-MoEalpha=0.5, distribution=non-IID2026.06 | 41.69 | — | — | — | — | — | — | — | — | 43.72 | 41.68 | |
| AWQModel=Llama-3.2-1B, Quantization=W3A162026.01 | 40.23 | — | — | — | — | — | — | — | — | — | — | |
| FedAvgalpha=1.0, distribution=non-IID2026.06 | 40.12 | — | — | — | — | — | — | — | — | 40.02 | 40.1 | |
| AWQModel=Llama-3-8B, Quantization=W3A162026.01 | 39.5 | — | — | — | — | — | — | — | — | — | — | |
| FedMoEalpha=0.5, distribution=non-IID2026.06 | 39.21 | — | — | — | — | — | — | — | — | 40.63 | 39.19 | |
| SpinQuantModel=Llama-3.2-1B, Quantization=W3A162026.01 | 39.1 | — | — | — | — | — | — | — | — | — | — | |
| PFL-MoEalpha=0.5, distribution=non-IID2026.06 | 37.89 | — | — | — | — | — | — | — | — | 39.47 | 37.4 | |
| FC-MoEalpha=0.1, distribution=non-IID2026.06 | 37.64 | — | — | — | — | — | — | — | — | 40.17 | 37.6 | |
| OmniQuantModel=Llama-3.2-1B, Quantization=W3A162026.01 | 36.5 | — | — | — | — | — | — | — | — | — | — | |
| FedMoEalpha=0.1, distribution=non-IID2026.06 | 35.06 | — | — | — | — | — | — | — | — | 36.99 | 35 | |
| FedAvgalpha=0.5, distribution=non-IID2026.06 | 34.82 | — | — | — | — | — | — | — | — | 38.54 | 34.78 | |
| PFL-MoEalpha=0.1, distribution=non-IID2026.06 | 33.25 | — | — | — | — | — | — | — | — | 35.41 | 33.01 | |
| FedAvgalpha=0.1, distribution=non-IID2026.06 | 30.08 | — | — | — | — | — | — | — | — | 38.7 | 30.02 |