Chinese Multitask Language Understanding on CMMLU
86.48AccuracyNTele-R1-32B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| NTele-R1-32B2025.08 | 86.48 | — | — | — | |
| DS-32B2025.08 | 84.88 | — | — | — | |
| Yi-34B + RTDevaluation=5-shot2024.09 | 81.8 | — | — | — | |
| Qwen2.5 (base)Params=7B, Tokens=18T, Complexity Type=Quadratic, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 81.73 | — | — | — | |
| Qwen2.5 (base)Params=7B, Tokens=18T, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 81.5 | — | — | — | |
| Yi-34Bevaluation=5-shot ICL2024.09 | 81 | — | — | — | |
| Qwen2.5Parameters=7B, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=generation-based2025.09 | 79.14 | — | — | — | |
| SpikingBrain-76B (76B-A12B)Params=12B/76B, Tokens=+160B, Complexity Type=Hybrid, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 78.83 | — | — | — | |
| SpikingBrain-76BParameters=12B/76B, Complexity Type=Hybrid, Evaluation Framework=vLLM, Evaluation Method=generation-based2025.09 | 77.41 | — | — | — | |
| Qwen3-4BParams=4B2025.12 | 77.01 | — | — | — | |
| Qwen2.5-3BParams=3B2025.12 | 73.92 | — | — | — | |
| Yi-34B + RTDevaluation=zero-shot2024.09 | 73.9 | — | — | — | |
| SpikingBrain-7BParams=7B, Tokens=+150B, Complexity Type=Linear, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 71.58 | — | — | — | |
| Qwen2-1.5BParams=1.5B2025.12 | 70.62 | — | — | — | |
| Yi-34Bevaluation=zero-shot2024.09 | 70.3 | — | — | — | |
| SpikingBrain-7BParameters=7B, Complexity Type=Linear, Evaluation Framework=vLLM, Evaluation Method=generation-based2025.09 | 68.76 | — | — | — | |
| Qwen2.5-1.5BParams=1.5B2025.12 | 68.01 | — | — | — | |
| HySparse# Shots=5-shot, Model Architecture=80B MoE (Hybrid 1:11), Attention Variant=HySparse2026.02 | 67 | — | — | — | |
| Full-Attn# Shots=5-shot, Model Architecture=80B MoE (Hybrid 1:11), Attention Variant=Full-Attn2026.02 | 66.7 | — | — | — | |
| Qwen3-1.7BParams=1.7B2025.12 | 66.55 | — | — | — | |
| HSA-ULTraining Strategy=Annealing, Architecture=MoE, Total Params=8B, Activated Params=1B, Training Tokens=8T2025.11 | 64.41 | — | — | — | |
| Gemma2-27BParams=27B, Tokens=13T, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 61.8 | — | — | — | |
| Hybrid SWA# Shots=5-shot, Model Architecture=80B MoE (Hybrid 1:11), Attention Variant=Hybrid SWA2026.02 | 58.4 | — | — | — | |
| TRM-MoETraining Strategy=Base, Architecture=MoE, Total Params=8B, Activated Params=1B, Training Tokens=8T2025.11 | 57.68 | — | — | — | |
| HSA-ULTraining Strategy=Base, Architecture=MoE, Total Params=8B, Activated Params=1B, Training Tokens=8T2025.11 | 57.49 | — | — | — | |
| Llama3Parameters=8B, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=generation-based2025.09 | 55.17 | — | — | — | |
| GLM3-6B + RTDevaluation=5-shot2024.09 | 54.7 | — | — | — | |
| GLM3-6Bevaluation=5-shot ICL2024.09 | 54.5 | — | — | — | |
| HySparse# Shots=5-shot, Model Architecture=7B Dense (Hybrid 1:3), Attention Variant=HySparse2026.02 | 54.5 | — | — | — | |
| GLM3-6B + RTDevaluation=zero-shot2024.09 | 53.3 | — | — | — | |
| LLama2-70bParams=70B, Tokens=2T, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 52.94 | — | — | — | |
| Hybrid SWA# Shots=5-shot, Model Architecture=7B Dense (Hybrid 1:3), Attention Variant=Hybrid SWA2026.02 | 52.9 | — | — | — | |
| Mixtral-8*7B (47B-A13B)Params=13B/47B, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 52.7 | — | — | — | |
| Full-Attn# Shots=5-shot, Model Architecture=7B Dense (Hybrid 1:3), Attention Variant=Full-Attn2026.02 | 52.5 | — | — | — | |
| Llama3.1Params=8B, Tokens=15T, Complexity Type=Quadratic, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 52.44 | — | — | — | |
| Qwen3-0.6BParams=0.6B2025.12 | 52.36 | — | — | — | |
| Qwen2.5Training Strategy=Annealing, Architecture=Dense, Total Params=0.5B, Activated Params=0.5B, Training Tokens=18T2025.11 | 52.1 | — | — | — | |
| Qwen3Training Strategy=Annealing, Architecture=Dense, Total Params=0.6B, Activated Params=0.6B, Training Tokens=36T2025.11 | 51.97 | — | — | — | |
| Jamba (52B-A12B)Params=12B/52B, Complexity Type=Hybrid, Evaluation Framework=vLLM, Evaluation Method=perplexity-based2025.09 | 51.11 | — | — | — | |
| MixtralParameters=13B/47B, Complexity Type=Quadratic, Evaluation Framework=vLLM, Evaluation Method=generation-based2025.09 | 51.03 | — | — | — | |
| SmolLM3-3BParams=3B2025.12 | 49.35 | — | — | — | |
| PCMind-2.1-Kaiyuan-2BParams=2B2025.12 | 49.25 | — | — | — | |
| GLM3-6Bevaluation=zero-shot2024.09 | 48.8 | — | — | — | |
| YuLan-Mini-2.4BParams=2.4B2025.12 | 48.14 | — | — | — | |
| MistralParams=7B, Complexity Type=Linear, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 44.58 | — | — | — | |
| llama-3.2-3BParams=3B2025.12 | 44.33 | — | — | — | |
| Falcon-MambaParams=7B, Tokens=5.8T, Complexity Type=Linear, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 42.5 | — | — | — | |
| HSA-ULTraining Strategy=Annealing, Architecture=Dense, Total Params=0.5B, Activated Params=0.5B, Training Tokens=4T2025.11 | 42.08 | — | — | — | |
| gemma2-2BParams=2B2025.12 | 39.63 | — | — | — | |
| Zamba-v1Params=7B, Tokens=1T, Complexity Type=Hybrid, Evaluation Framework=HuggingFace, Evaluation Protocol=perplexity-based2025.09 | 38.42 | — | — | — | |
| SmolLM2-1.7BParams=1.7B2025.12 | 34.03 | — | — | — | |
| Llama 2 7B (baseline)Base Model=Llama 2 7B, Compression Ratio=0%2025.05 | 31.8 | — | — | — | |
| DenseCompression Ratio=0%, Post-training compensation=No, Evaluation Protocol=Zero-shot2024.12 | 31.8 | — | — | — | |
| LLM-Streamline-FFNCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 31.7 | — | — | — | |
| llama-3.2-1BParams=1B2025.12 | 31.03 | — | — | — | |
| GRASPCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 30.7 | — | — | — | |
| ReplaceMe (Cosine)Train-Free=true, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 30.4 | — | — | — | |
| LLM-StreamlineTrain-Free=false, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 29.4 | — | — | — | |
| LLM-Streamline-LayerCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 29.4 | — | — | — | |
| ReplaceMe (LS)Train-Free=true, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 29.2 | — | — | — | |
| UIDLTrain-Free=false, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 28.9 | — | — | — | |
| ShortGPTCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 28.9 | — | — | — | |
| OLMo-2-0425-1BParams=1B2025.12 | 28.62 | — | — | — | |
| LaCoTrain-Free=false, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 25.2 | — | — | — | |
| LaCoCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 25.2 | — | — | — | |
| LLMPrunerTrain-Free=false, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 25 | — | — | — | |
| LLMPrunerCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 25 | — | — | — | |
| SliceGPTTrain-Free=false, Base Model=Llama 2 7B, Compression Ratio=25%2025.05 | 24.8 | — | — | — | |
| SliceGPTCompression Ratio=25%, Post-training compensation=Yes, Evaluation Protocol=Zero-shot2024.12 | 24.8 | — | — | — | |
| AdamWModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | — | 64.64 | — | — | |
| AquilaChat2Size=34B2024.03 | — | — | 67.5 | 70 | |
| Baichuan2-ChatSize=13B2024.03 | — | — | 58.6 | 59.5 | |
| InternLM-ChatSize=20B2024.03 | — | — | 53.6 | 53.8 | |
| LLaMA2-ChatSize=13B2024.03 | — | — | 27.5 | 35.1 | |
| LLaMA2-ChatSize=70B2024.03 | — | — | 36.1 | 41 | |
| Mamba-2.8BBackbone=Mamba-2.8B, Pruning ratio=0%2024.03 | — | 25.32 | — | — | |
| MONAModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | — | 67.56 | — | — | |
| MuonModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | — | 67.23 | — | — | |
| Qwen-ChatSize=14B2024.03 | — | — | 67.7 | 70.6 | |
| RWKV-7BBackbone=RWKV-7B, Pruning ratio=0%2024.03 | — | 28.54 | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=10.9%2024.03 | — | 25 | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=20.3%2024.03 | — | 25.37 | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=25%2024.03 | — | 24.89 | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=31.3%2024.03 | — | 24.77 | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=9.4%2024.03 | — | 25.03 | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=18.8%2024.03 | — | 25 | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=25%2024.03 | — | 25.08 | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=28.1%2024.03 | — | 25.15 | — | — | |
| Yi-ChatSize=6B2024.03 | — | — | 69.4 | 74.7 | |
| Yi-ChatSize=34B2024.03 | — | — | 79.1 | 81.3 | |
| Yi-Chat-4bits(AWQ)Size=6B2024.03 | — | — | 67.7 | 73.3 | |
| Yi-Chat-4bits(AWQ)Size=34B2024.03 | — | — | 78.2 | 80.5 | |
| Yi-Chat-8bits(GPTQ)Size=6B2024.03 | — | — | 69.2 | 74.7 | |
| Yi-Chat-8bits(GPTQ)Size=34B2024.03 | — | — | 79.1 | 81.2 |