Massive Multitask Language Understanding on MMLU
83.34AccuracyLoPT-GRPO
Evaluation results
| Method | Links | |
|---|---|---|
| LoPT-GRPOBackbone=Qwen2.5-32B2026.05 | 83.34 | |
| BaseBackbone=Qwen2.5-32B2026.05 | 83.32 | |
| E2E-GRPOBackbone=Qwen2.5-32B2026.05 | 83.24 | |
| Qwen3-8B-Instruct2026.04 | 81.7 | |
| MiniCPM-o 4.52026.04 | 77 | |
| SFTBackbone=Llama3.1 8B2026.03 | 69.49 | |
| SFTBackbone=Mistral 7B2026.03 | 67.32 | |
| PermLMBackbone=Llama3.1 8B2026.03 | 67 | |
| FP16Avg. Bits=16.00, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot2025.08 | 65.24 | |
| MoE + L_ERCNumber of Parameters=15B, Activated Parameters=700M, Loss Function=ERC loss2025.12 | 64.6 | |
| MoENumber of Parameters=15B, Activated Parameters=700M2025.12 | 63.2 | |
| Base ModelModel=Llama 3.1-8B2025.05 | 63 | |
| CoABackbone=Llama3.1 8B2026.03 | 62.87 | |
| Mistral-7BRatio=0%, Evaluation Protocol=5-shot2026.03 | 62.5 | |
| BaseBackbone=Llama3.1 8B2026.03 | 62 | |
| CASTModel=Llama 3.1-8B2025.05 | 61.9 | |
| MicroMixAvg. Bits=5.51, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 60.9 | |
| WASModel=Llama 3.1-8B2025.05 | 60.8 | |
| Original model (M_original)Model Family=qwen-2.5-1.5b-it2026.03 | 60.13 | |
| ITIModel=Llama 3.1-8B2025.05 | 60.1 | |
| PKU-RLHF aligned model (M_PKU-RLHF)Model Family=qwen-2.5-1.5b-it2026.03 | 60.08 | |
| Safety-Reset model (M_base)Model Family=qwen-2.5-1.5b-it2026.03 | 59.98 | |
| Base modelArchitecture=Mistral-7B2026.01 | 59.6 | |
| SFTBackbone=Qwen3 1.7B2026.03 | 59.45 | |
| ACTModel=Llama 3.1-8B2025.05 | 59.4 | |
| FlatQuantAvg. Bits=4.19, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 59.34 | |
| Self-MOA aligned model (M_Self-MOA)Model Family=qwen-2.5-1.5b-it2026.03 | 59.02 | |
| ShapLoRABackbone=LlaMA-3 8B2026.01 | 58.7 | |
| RAbArchitecture=Mistral-7B, Concept Direction=random2026.01 | 58.7 | |
| Base modelArchitecture=Zephyr-7B2026.01 | 58.4 | |
| RAbArchitecture=Zephyr-7B, Concept Direction=random2026.01 | 57.7 | |
| QUIKAvg. Bits=5.95, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 57.63 | |
| Original model (M_original)Model Family=gemma-2-2b-it2026.03 | 57.51 | |
| AutoLoRABackbone=LlaMA-3 8B2026.01 | 57.1 | |
| Base ModelModel=Mistral 7B2025.05 | 57.1 | |
| AtomAvg. Bits=4.25, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 57.1 | |
| CoABackbone=Mistral 7B2026.03 | 57 | |
| INT6Avg. Bits=6.00, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 56.82 | |
| MOELoRABackbone=LlaMA-3 8B2026.01 | 56.6 | |
| CoABackbone=Qwen3 1.7B2026.03 | 56.35 | |
| Base (no adapt.)Backbone=Mistral-7B-Instruct2026.04 | 56.3 | |
| PermLMBackbone=Qwen3 1.7B2026.03 | 56 | |
| RAdArchitecture=Zephyr-7B, Concept Direction=random2026.01 | 55.9 | |
| SafeAnchorBackbone=Mistral-7B-Instruct2026.04 | 55.9 | |
| Self-MOA aligned model (M_Self-MOA)Model Family=gemma-2-2b-it2026.03 | 55.8 | |
| SafeGrad + LoRABackbone=Mistral-7B-Instruct2026.04 | 55.6 | |
| DenseTraining Tokens=100B, Inference Top-K=N/A2026.01 | 55.53 | |
| ExtraBackbone=Llama3.1 8B2026.03 | 55.5 | |
| ITIModel=Mistral 7B2025.05 | 55.4 | |
| EWC + LoRABackbone=Mistral-7B-Instruct2026.04 | 55.4 | |
| DenseTraining Tokens=100B, Inference Top-K=22026.01 | 55.34 | |
| InfLLM v2Training Tokens=100B, Inference Top-K=N/A, Training Top-K=42026.01 | 55.33 | |
| Safety InterleavingBackbone=Mistral-7B-Instruct2026.04 | 55.3 | |
| PHSATraining Tokens=100B, Inference Top-K=N/A, Training Top-K=22026.01 | 55.26 | |
| PHSATraining Tokens=100B, Inference Top-K=4, Training Top-K=42026.01 | 55.11 | |
| PHSATraining Tokens=100B, Inference Top-K=N/A, Training Top-K=42026.01 | 55.1 | |
| Safe LoRABackbone=Mistral-7B-Instruct2026.04 | 55.1 | |
| PHSATraining Tokens=100B, Inference Top-K=2, Training Top-K=22026.01 | 55 | |
| PermLMBackbone=Mistral 7B2026.03 | 55 | |
| Safety-Reset model (M_base)Model Family=gemma-2-2b-it2026.03 | 54.99 | |
| InfLLM v2Training Tokens=100B, Inference Top-K=N/A, Training Top-K=22026.01 | 54.98 | |
| InfLLM v2Training Tokens=100B, Inference Top-K=4, Training Top-K=42026.01 | 54.97 | |
| PKU-RLHF aligned model (M_PKU-RLHF)Model Family=gemma-2-2b-it2026.03 | 54.97 | |
| RAdArchitecture=Zephyr-7B, Concept Direction=truthfulness2026.01 | 54.9 | |
| CASTModel=Mistral 7B2025.05 | 54.9 | |
| Standard LoRABackbone=Mistral-7B-Instruct2026.04 | 54.9 | |
| InfLLM v2Training Tokens=100B, Inference Top-K=2, Training Top-K=22026.01 | 54.79 | |
| WASModel=Mistral 7B2025.05 | 54.2 | |
| ACTModel=Mistral 7B2025.05 | 54 | |
| RAdArchitecture=Mistral-7B, Concept Direction=random2026.01 | 53.6 | |
| ExtraBackbone=Mistral 7B2026.03 | 53.57 | |
| QuaRotAvg. Bits=4.12, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 53.19 | |
| AMXFP4Avg. Bits=5.00, Base Model=Llama3.1-8B, Evaluation Protocol=5-shot, KV Cache Quantization=Asymmetric INT4 (group size 64)2025.08 | 53.11 | |
| RAdArchitecture=Mistral-7B, Concept Direction=truthfulness2026.01 | 53 | |
| SASFTModel=Qwen3-8B-Base2025.07 | 52.88 | |
| RAbArchitecture=Zephyr-7B, Concept Direction=truthfulness2026.01 | 52 | |
| RAbArchitecture=Mistral-7B, Concept Direction=truthfulness2026.01 | 50.2 | |
| BaseBackbone=Qwen3 1.7B2026.03 | 50 | |
| SFTModel=Qwen3-8B-Base2025.07 | 49.67 | |
| BaseBackbone=Mistral 7B2026.03 | 49 | |
| SFT+PenaltyModel=Qwen3-8B-Base2025.07 | 47.68 | |
| SFTModel=Gemma-2-9B2025.07 | 47.56 | |
| SFT+GRPOModel=Gemma-2-9B2025.07 | 47.47 | |
| SLEB-pruned LLaMA2-7BSparsity=10%, Model (GB)=11.0, Bits=162026.02 | 46.94 | |
| SASFTModel=Gemma-2-9B2025.07 | 46.85 | |
| SFT+PenaltyModel=Gemma-2-9B2025.07 | 46.66 | |
| ExtraBackbone=Qwen3 1.7B2026.03 | 46.27 | |
| PKU-RLHF aligned model (M_PKU-RLHF)Model Family=llama-3.2-1b-it2026.03 | 46.12 | |
| Original model (M_original)Model Family=llama-3.2-1b-it2026.03 | 46.07 | |
| Self-MOA aligned model (M_Self-MOA)Model Family=llama-3.2-1b-it2026.03 | 45.77 | |
| Safety-Reset model (M_base)Model Family=llama-3.2-1b-it2026.03 | 45.35 | |
| SFT+GRPOModel=Qwen3-8B-Base2025.07 | 45.27 | |
| SLEB-pruned LLaMA2-7BSparsity=10%, Model (GB)=2.9, Bits=42026.02 | 44.87 | |
| SoLARatio=20%, Evaluation Protocol=5-shot2026.03 | 44.2 | |
| SLEB-pruned LLaMA2-7BSparsity=20%, Model (GB)=9.9, Bits=162026.02 | 44.06 | |
| SLEB-pruned LLaMA2-7BSparsity=20%, Model (GB)=2.6, Bits=42026.02 | 42.35 | |
| SVD-LLMRatio=20%, Evaluation Protocol=5-shot2026.03 | 41.8 | |
| Self-MOA aligned model (M_Self-MOA)Model Family=gemma-3-1b-it2026.03 | 39.82 | |
| PKU-RLHF aligned model (M_PKU-RLHF)Model Family=gemma-3-1b-it2026.03 | 39.23 | |
| Safety-Reset model (M_base)Model Family=gemma-3-1b-it2026.03 | 39.19 |