Reading Comprehension Accuracy on RACE
89.95AccuracyQwen-1.5 14B
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen-1.5 14BRole=Teacher2024.07 | 89.95 | |
| Qwen-1.5 14BModel Type=Teacher2024.07 | 89.95 | |
| Qwen-1.5 7BRole=Teacher2024.07 | 85.78 | |
| Qwen-1.5 7BRole=Teacher2024.07 | 85.78 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 82.53 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 82.53 | |
| PoEBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 81.39 | |
| ALBMBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 80.69 | |
| DIRBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 80.32 | |
| LPBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 80.3 | |
| Qwen-1.5 4BRole=Student2024.07 | 80.17 | |
| Qwen-1.5 4BModel Type=Student2024.07 | 80.17 | |
| KDTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 80.13 | |
| KDTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 80.13 | |
| CPTTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 79.56 | |
| CPTTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 79.56 | |
| MiniLLMTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 79.22 | |
| MiniLLMTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 4B2024.07 | 79.22 | |
| BaseBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 79.21 | |
| LPBackbone=Llama3.1-8B-Instruct2025.12 | 78.9 | |
| SKBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 78.82 | |
| InfoRMBackbone=OpenRLHF-Llama3-8B-SFT2025.12 | 78.72 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 1.8B, Training Tokens=15B2024.07 | 71.5 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 1.8B, Training Tokens=30B2024.07 | 71.4 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 1.8B, Training Tokens=20B2024.07 | 71.27 | |
| DDKBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 70.98 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 1.8B, Training Tokens=10B2024.07 | 70.84 | |
| KDBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 69.86 | |
| Qwen-1.5 1.8BRole=Student2024.07 | 69.57 | |
| Qwen-1.5 1.8BRole=Student2024.07 | 69.57 | |
| DDKTeacher Model=Qwen-1.5 14B, Student Model=Qwen-1.5 1.8B, Training Tokens=5B2024.07 | 69.4 | |
| MiniLLMBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 69.1 | |
| CPTBase Model=Qwen-1.5 1.8B, Teacher=Qwen-1.5 7B2024.07 | 69 | |
| BaseBackbone=Llama3.1-8B-Instruct2025.12 | 66.5 | |
| InfoRMBackbone=Llama3.1-8B-Instruct2025.12 | 65.2 | |
| DIRBackbone=Llama3.1-8B-Instruct2025.12 | 62.02 | |
| PoEBackbone=Llama3.1-8B-Instruct2025.12 | 60.03 | |
| MixtralParameters=8x22B2024.07 | 59.2 | |
| Llama 3Parameters=70B2024.07 | 59 | |
| ALBMBackbone=Llama3.1-8B-Instruct2025.12 | 59 | |
| Llama 3Parameters=405B2024.07 | 58.1 | |
| Llama 3Parameters=8B2024.07 | 54.3 | |
| SKBackbone=Llama3.1-8B-Instruct2025.12 | 53.89 | |
| MistralParameters=7B2024.07 | 53 | |
| SD-MoEModel Scale=Large (7B-1.5B), Base Model=Qwen, Method Configuration=SD-MoE2026.02 | 51.48 | |
| QwenModel Scale=Large (7B-1.5B), Base Model=Qwen, Method Configuration=Baseline2026.02 | 50.65 | |
| SD-MoEModel Scale=Small (2B-0.8B), Base Model=Qwen, Method Configuration=SD-MoE2026.02 | 50.61 | |
| DeepSeekModel Scale=Small (2B-0.8B), Base Model=DeepSeek, Method Configuration=Baseline2026.02 | 50.51 | |
| SD-MoEModel Scale=Small (2B-0.8B), Base Model=DeepSeek, Method Configuration=SD-MoE2026.02 | 50.06 | |
| GemmaParameters=7B2024.07 | 48.8 | |
| QwenModel Scale=Small (2B-0.8B), Base Model=Qwen, Method Configuration=Baseline2026.02 | 48.54 | |
| Nemotron-4 15B + MaskLLMBackbone=Nemotron-4 15B, Pruning Method=MaskLLM, Sparsity Pattern=2:42024.09 | 45.45 | |
| LLaMA-2 7B (Dense)Backbone=LLaMA-2 7B, Pruning Method=Dense, Sparsity Pattern=None2024.09 | 44.11 | |
| LLaMA-2 13B + MaskLLMBackbone=LLaMA-2 13B, Pruning Method=MaskLLM, Sparsity Pattern=2:42024.09 | 41.24 | |
| LLaMA-2 7B + MaskLLMBackbone=LLaMA-2 7B, Pruning Method=MaskLLM, Sparsity Pattern=2:42024.09 | 40.77 | |
| Llama3 8BCompression Ratio=Baseline2025.09 | 40.3 | |
| Llama3 8B2026.02 | 40.3 | |
| COMPOTCR=0.22026.02 | 40.1 | |
| Llama 3.2 3BAttn CR=N/A2025.08 | 40 | |
| Matrix PCABase Model=Llama 3.1 8B, Attn CR=20%2025.08 | 40 | |
| SVD-LLMBase Model=Llama 3.1 8B, Attn CR=20%2025.08 | 39.6 | |
| Matrix PCABase Model=Llama 3.2 3B, Attn CR=20%2025.08 | 39.3 | |
| Llama 3.1 8BAttn CR=N/A2025.08 | 39.3 | |
| Llama2 7BCR=None2025.09 | 39.2 | |
| CoSpaDi (per-layer)CR=0.22025.09 | 39 | |
| SVD-LLMBase Model=Llama 3.2 3B, Attn CR=20%2025.08 | 38.6 | |
| COMPOTCR=0.32026.02 | 38.4 | |
| CoSpaDi (grouped)CR=0.22025.09 | 38.3 | |
| CoSpaDiCompression Ratio=0.22025.09 | 38.2 | |
| DLMTokens=100B, Model Scale=1.5B, Initialization=Autoregressive LLM (Qwen2.5)2026.01 | 37.8 | |
| Llama 3.2 1BAttn CR=N/A2025.08 | 37.8 | |
| DLM + extra tokenTokens=100B, Model Scale=1.5B, Initialization=Autoregressive LLM (Qwen2.5)2026.01 | 37.61 | |
| DLM + GATokens=100B, Model Scale=1.5B, Initialization=Autoregressive LLM (Qwen2.5)2026.01 | 37.42 | |
| SVD-LLMBase Model=Llama 3.2 1B, Attn CR=20%2025.08 | 37.3 | |
| CoSpaDi (per-layer)CR=0.32025.09 | 36.8 | |
| SVD-LLMCR=0.22025.09 | 36.6 | |
| COMPOTCR=0.42026.02 | 36.6 | |
| SVD-LLMBase Model=Llama 3.2 1B, Attn CR=30%2025.08 | 36.5 | |
| CoSpaDiCompression Ratio=0.32025.09 | 36.2 | |
| CoSpaDi (grouped)CR=0.32025.09 | 35.9 | |
| Matrix PCABase Model=Llama 3.2 1B, Attn CR=20%2025.08 | 35.6 | |
| ReplaceMeCompression Ratio=0.222025.09 | 35.4 | |
| ReplaceMeCR=0.222026.02 | 35.4 | |
| LLM-PrunerCompression Ratio=0.22025.09 | 35.1 | |
| LLM-PrunerCR=0.22026.02 | 35.1 | |
| Basis SharingCR=0.22025.09 | 34.9 | |
| SVD-LLMCR=0.32025.09 | 34.7 | |
| Matrix PCABase Model=Llama 3.2 1B, Attn CR=30%2025.08 | 34.2 | |
| DLM + GATokens=30B, Model Scale=0.5B, Initialization=Autoregressive LLM (Qwen2.5)2026.01 | 34.07 | |
| ReplaceMeCompression Ratio=0.312025.09 | 34 | |
| ReplaceMeCR=0.312026.02 | 34 | |
| GPT3 2B + MaskLLMBackbone=GPT3 2B, Pruning Method=MaskLLM, Sparsity Pattern=2:42024.09 | 33.88 | |
| DLM + extra tokenTokens=30B, Model Scale=0.5B, Initialization=Autoregressive LLM (Qwen2.5)2026.01 | 33.79 | |
| Qwen3-0.6BCompression Ratio=None2026.02 | 33.7 | |
| DLM + GATokens=100B, Initialization=from scratch, Model Scale=0.5B parameters2026.01 | 33.68 | |
| CoSpaDi (per-layer)CR=0.42025.09 | 33.4 | |
| CoSpaDi (grouped)CR=0.42025.09 | 33.2 | |
| COMPOT†Compression Ratio=0.22026.02 | 33 | |
| DLM + extra tokenTokens=100B, Initialization=from scratch, Model Scale=0.5B parameters2026.01 | 32.73 | |
| COMPOTCompression Ratio=0.32026.02 | 32.7 |