Multitask Knowledge on MMLU
76.5AccuracyBase
Evaluation results
| Method | Links | |
|---|---|---|
| Basebackbone=DS-8B2025.09 | 76.5 | |
| IPObackbone=DS-8B2025.09 | 74.7 | |
| RealSafebackbone=DS-8B, alignment=SFT-based2025.09 | 72.7 | |
| BaselineModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 70.2 | |
| Expanded DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 70.2 | |
| Token DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 69.1 | |
| Expert DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 66.2 | |
| Model-GLUEModel Zoo=Mistral2024.10 | 64.39 | |
| F-L-SModel Zoo=Mistral2024.10 | 63.42 | |
| DAREModel Zoo=Mistral2024.10 | 63.27 | |
| KALEBackbone=Llama3 8B2026.01 | 63.27 | |
| Best Single ModelModel Zoo=Mistral2024.10 | 61.77 | |
| COT-DistillBackbone=Llama3 8B2026.01 | 60.94 | |
| TIESModel Zoo=Mistral2024.10 | 60.9 | |
| Qwen3-1.7BType=Base Model2026.05 | 60.61 | |
| PDBackbone=Llama3 8B2026.01 | 60.17 | |
| LoRAFine-tuning dataset=MetaMathQA2026.05 | 60.12 | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open2026.05 | 60.11 | |
| LoRA-NullFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy, Calibration model size=4B2026.05 | 60.07 | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=4B2026.05 | 60.06 | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=1.7B2026.05 | 59.97 | |
| LoRA-NullFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy2026.05 | 59.96 | |
| OPLoRAFine-tuning dataset=MetaMathQA2026.05 | 59.94 | |
| MiLoRAFine-tuning dataset=MetaMathQA2026.05 | 59.9 | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=0.6B2026.05 | 59.86 | |
| BaselineModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 59.8 | |
| Token DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 59.3 | |
| Expanded DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 59.3 | |
| Distill-SBSBackbone=Llama3 8B2026.01 | 58.72 | |
| Expert DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 58.1 | |
| BaselineModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 57.3 | |
| Token DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 57.3 | |
| Expanded DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 57.2 | |
| CorDAFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy, Calibration model size=4B2026.05 | 57.11 | |
| CorDAFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy2026.05 | 56.84 | |
| FP16Backbone=LLaMA-3.2 3B2026.03 | 54.06 | |
| Expert DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 53.3 | |
| BaselineModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 52.8 | |
| Expanded DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 52.3 | |
| FP16Backbone=LLaMA-2 13B2026.03 | 52.04 | |
| Token DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 51.5 | |
| Expert DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 50.6 | |
| FlatQuantBackbone=LLaMA-3.2 3B, Calib. Time=∼76m2026.03 | 50 | |
| Teacher (GPT-OSS-20B)Evaluation Mode=completion mode2026.05 | 49.6 | |
| FlatQuantBackbone=LLaMA-2 13B, Calib. Time=∼225m2026.03 | 49.59 | |
| DuQuantBackbone=LLaMA-2 13B, Calib. Time=∼486m2026.03 | 48.59 | |
| SpinQuantBackbone=LLaMA-2 13B, Calib. Time=∼721m2026.03 | 47.8 | |
| OstQuantBackbone=LLaMA-2 13B, Calib. Time=∼92m2026.03 | 47.77 | |
| QuaRotBackbone=LLaMA-2 13B, Calib. Time=∼44m2026.03 | 47.25 | |
| SERQBackbone=LLaMA-2 13B, Calib. Time=∼48m2026.03 | 47.17 | |
| Nemotron-CC_ASI+Pre-training Curation Strategy=AI Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 46.13 | |
| OstQuantBackbone=LLaMA-3.2 3B, Calib. Time=∼26m2026.03 | 46.05 | |
| SERQBackbone=LLaMA-3.2 3B, Calib. Time=∼15m2026.03 | 45.7 | |
| DuQuantBackbone=LLaMA-3.2 3B, Calib. Time=∼140m2026.03 | 44.79 | |
| QuaRotBackbone=LLaMA-3.2 3B, Calib. Time=∼8m2026.03 | 44.75 | |
| SpinQuantBackbone=LLaMA-3.2 3B, Calib. Time=∼156m2026.03 | 42.42 | |
| FP16Backbone=LLaMA-2 7B2026.03 | 41.83 | |
| FlatQuantBackbone=LLaMA-2 7B, Calib. Time=∼131m2026.03 | 38.24 | |
| OstQuantBackbone=LLaMA-2 7B, Calib. Time=∼72m2026.03 | 37.39 | |
| SERQBackbone=LLaMA-2 7B, Calib. Time=∼23m2026.03 | 37.03 | |
| Warmup-Stable-Only (WSO)Model=1B, Pre-training Scheduler=Warmup-Stable-Only (WSO), alpha_pre=1.0, alpha_mid=1.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 36.6 | |
| WSDModel=1B, Pre-training Scheduler=WSD, alpha_pre=0.1, alpha_mid=0.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 36 | |
| WSDModel=1B, Pre-training Scheduler=WSD, alpha_pre=0.1, alpha_mid=1.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 35.9 | |
| CosineModel=1B, Pre-training Scheduler=Cosine, alpha_pre=0.1, alpha_mid=0.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 35.9 | |
| WSDModel=1B, Pre-training Scheduler=WSD, alpha_pre=1.0, alpha_mid=0.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 35.5 | |
| CosineModel=1B, Pre-training Scheduler=Cosine, alpha_pre=0.1, alpha_mid=1.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 35.4 | |
| LinearModel=1B, Pre-training Scheduler=Linear, alpha_pre=0.1, alpha_mid=1.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 35 | |
| SpinQuantBackbone=LLaMA-2 7B, Calib. Time=∼598m2026.03 | 34.8 | |
| LinearModel=1B, Pre-training Scheduler=Linear, alpha_pre=0.1, alpha_mid=0.0, Pre-training tokens=2T, Mid-training tokens=500B, Training Stage=Supervised fine-tuned (SFT)2026.03 | 34.8 | |
| QuaRotBackbone=LLaMA-2 7B, Calib. Time=∼31m2026.03 | 33.58 | |
| DuQuantBackbone=LLaMA-2 7B, Calib. Time=∼255m2026.03 | 33.46 | |
| Nemotron-CC_ASIPre-training Curation Strategy=AI Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 32.55 | |
| DCLMPre-training Curation Strategy=Human Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 28.54 | |
| Fineweb-EduPre-training Curation Strategy=Human Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 28.38 | |
| Nemotron-CCPre-training Curation Strategy=AI Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 27.49 | |
| Ultra-FinewebPre-training Curation Strategy=Human Discovered, Model Parameters=3B, Training Token Budget=500B tokens2026.03 | 25.53 | |
| ACPScoring=ACP, K=82026.05 | 23.7 | |
| DO-ACPScoring=DO-ACP, K=42026.05 | 23.7 | |
| CPScoring=CP, K=42026.05 | 23.6 | |
| SFScoring=SF, K=42026.05 | 23.3 | |
| ACPScoring=ACP, K=42026.05 | 23.3 | |
| DO-ACPScoring=DO-ACP, K=82026.05 | 23.3 | |
| Random FFN + teacher attn2026.05 | 23.2 | |
| CPScoring=CP, K=82026.05 | 23.1 | |
| Random initialization2026.05 | 23 | |
| SFScoring=SF, K=82026.05 | 22.8 | |
| DualAlpha (α)=63/64, Data repetitions=12025.12 | 4.9 | |
| AutoregressiveAlpha (α)=1, Data repetitions=12025.12 | 3.8 | |
| AutoregressiveAlpha (α)=1, Data repetitions=322025.12 | 3.8 | |
| DualAlpha (α)=3/4, Data repetitions=322025.12 | 3.6 | |
| DualAlpha (α)=1/8, Data repetitions=1282025.12 | 1.6 | |
| AutoregressiveAlpha (α)=1, Data repetitions=1282025.12 | 1.1 |