Language Understanding on MMLU (5-shot)
90.58AccuracyERNIE 5.0-Base
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| ERNIE 5.0-BaseNumber of shots=5-shot2026.02 | 90.58 | — | — | — | — | — | — | — | |
| DS V3.2-Exp-BaseNumber of shots=5-shot2026.02 | 88.6 | — | — | — | — | — | — | — | |
| Kimi K2-BaseNumber of shots=5-shot2026.02 | 88.4 | — | — | — | — | — | — | — | |
| Dream-v0-7B-InstructParallel Decoding=false2026.05 | 73.34 | — | — | — | — | — | 9.97 | 1 | |
| dLLM-CacheAcceleration Method=dLLM-Cache, Parallel Decoding=false2026.05 | 73.08 | — | — | — | — | — | 19.44 | 1.95 | |
| dKV-CacheAcceleration Method=dKV-Cache, Parallel Decoding=false2026.05 | 72.93 | — | — | — | — | — | 16.75 | 1.68 | |
| Fast-dLLMAcceleration Method=Fast-dLLM, Parallel Decoding=false2026.05 | 72.14 | — | — | — | — | — | 18.74 | 1.88 | |
| Dynamic-dLLMAcceleration Method=Dynamic-dLLM, Parallel Decoding=true2026.05 | 72.1 | — | — | — | — | — | 31.31 | 3.14 | |
| Dynamic-dLLMAcceleration Method=Dynamic-dLLM, Parallel Decoding=false2026.05 | 72.09 | — | — | — | — | — | 25.82 | 2.59 | |
| Fast-dLLMAcceleration Method=Fast-dLLM, Parallel Decoding=true2026.05 | 71.97 | — | — | — | — | — | 30.11 | 3.02 | |
| fp16Size=70B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 69.2 | 65.2 | 58.3 | 80.5 | 75.2 | — | — | — | |
| OmniQuantSize=70B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 67.3 | 64.1 | 55.2 | 78.5 | 73.1 | — | — | — | |
| AstroSize=70B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 66.9 | 64.2 | 54.7 | 77.9 | 72.2 | — | — | — | |
| MagRSize=70B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 66.1 | 63 | 53.9 | 77 | 72.4 | — | — | — | |
| FP16Backbone=LLaMA-3.1-8B, Quantization=None, Protocol=5-shot2026.02 | 65.3 | 59.8 | 56 | 76.4 | 72.2 | — | — | — | |
| FP16Model=LLaMA-3-8B, Quantization=FP162026.04 | 65.3 | — | — | — | — | — | — | — | |
| COVERCALModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 64.6 | — | — | — | — | — | — | — | |
| Max-ActVarModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 63.8 | — | — | — | — | — | — | — | |
| StratifiedModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 63.5 | — | — | — | — | — | — | — | |
| Max-PPLModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 63.4 | — | — | — | — | — | — | — | |
| RandomModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 63.1 | — | — | — | — | — | — | — | |
| LORABackbone Model=Mistral-v0.1 7B, Precision=16-bit2024.02 | 62.9 | — | — | — | — | — | — | — | |
| Pre-trainedBackbone Model=Mistral-v0.1 7B, Precision=16-bit2024.02 | 62.6 | — | — | — | — | — | — | — | |
| FP16Model=Mistral-7B, Quantization=FP162026.04 | 62.5 | — | — | — | — | — | — | — | |
| COVERCALModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 62 | — | — | — | — | — | — | — | |
| L4QModel=Mistral-v0.1 7B, Precision=4-bit2024.02 | 61.4 | — | — | — | — | — | — | — | |
| Max-ActVarModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 61.3 | — | — | — | — | — | — | — | |
| StratifiedModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 61.1 | — | — | — | — | — | — | — | |
| Max-PPLModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 61 | — | — | — | — | — | — | — | |
| RandomModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 60.8 | — | — | — | — | — | — | — | |
| LORABackbone Model=LLaMA-1 33B, Precision=16-bit2024.02 | 57.6 | — | — | — | — | — | — | — | |
| L4QModel=LLaMA-1 33B, Precision=4-bit2024.02 | 56.7 | — | — | — | — | — | — | — | |
| Pre-trainedBackbone Model=LLaMA-1 33B, Precision=16-bit2024.02 | 56.4 | — | — | — | — | — | — | — | |
| 32-bit AdamWModel=LLaMA-33B, Status=Fine-tuned2023.09 | 56.4 | — | — | — | — | — | — | — | |
| AstroBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 56.3 | 50.4 | 49.7 | 65.6 | 62.6 | — | — | — | |
| L4QBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 56.2 | — | — | — | — | — | — | — | |
| L4QModel=Mistral-v0.1 7B, Precision=3-bit2024.02 | 56.2 | — | — | — | — | — | — | — | |
| OmniQBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 55.9 | — | — | — | — | — | — | — | |
| FP16Model=Vicuna-v1.5-13B, Quantization=None2024.06 | 55.78 | 51.97 | 44.96 | 65.26 | 62.4 | — | — | — | |
| LORABackbone Model=LLaMA-2 13B, Precision=16-bit2024.02 | 55.7 | — | — | — | — | — | — | — | |
| LLaMA2-13bBit=16, Foundation Model=LLaMA2-13b2024.05 | 55.7 | 62.9 | 44.4 | 63.9 | 56.7 | — | — | — | |
| fp16Size=13B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 55.4 | 53.8 | 44.4 | 63.5 | 61.3 | — | — | — | |
| L4QModel=LLaMA-2 13B, Precision=4-bit2024.02 | 55.2 | — | — | — | — | — | — | — | |
| Original LLaMAModel=LLaMA-33B, Status=Zero-shot baseline2023.09 | 54.9 | — | — | — | — | — | — | — | |
| 4-bit AdamWModel=LLaMA-33B, Status=Fine-tuned2023.09 | 54.9 | — | — | — | — | — | — | — | |
| GPTQBit=4, Foundation Model=LLaMA2-13b2024.05 | 54.9 | 61.3 | 43.3 | 62.5 | 57.2 | — | — | — | |
| Pre-trainedBackbone Model=LLaMA-2 13B, Precision=16-bit2024.02 | 54.8 | — | — | — | — | — | — | — | |
| QA-LoRABit=4, Foundation Model=LLaMA2-13b2024.05 | 54.2 | 59.6 | 42.7 | 62.2 | 57.4 | — | — | — | |
| QAT-LORABackbone Model=Mistral-v0.1 7B, Precision=3&16-bit2024.02 | 54 | — | — | — | — | — | — | — | |
| QLORABackbone Model=Mistral-v0.1 7B, Precision=3&16-bit2024.02 | 53.6 | — | — | — | — | — | — | — | |
| LLM-QFABit=4, Foundation Model=LLaMA2-13b2024.05 | 53.4 | 59.1 | 42.1 | 61.1 | 56.2 | — | — | — | |
| L4QBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 53.1 | — | — | — | — | — | — | — | |
| L4QModel=LLaMA-1 33B, Precision=3-bit2024.02 | 53.1 | — | — | — | — | — | — | — | |
| OmniQBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 52.4 | — | — | — | — | — | — | — | |
| AstroSize=13B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 52.2 | 48.5 | 43.1 | 60.4 | 58.8 | — | — | — | |
| DuQuantModel=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 51.96 | 48.78 | 40.92 | 60.42 | 57.71 | — | — | — | |
| GPTQBackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 51.9 | — | — | — | — | — | — | — | |
| MagRBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 51.9 | 46.1 | 46.1 | 59.7 | 58.9 | — | — | — | |
| LLM-QFABit=3, Foundation Model=LLaMA2-13b2024.05 | 51.8 | 56.7 | 40.6 | 59.9 | 54.5 | — | — | — | |
| QA-LORABackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 51.7 | — | — | — | — | — | — | — | |
| QA-LoRABit=3, Foundation Model=LLaMA2-13b2024.05 | 51.7 | 56.8 | 41.9 | 59 | 53.5 | — | — | — | |
| OmniQuantBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=5-shot2026.02 | 51.7 | 45.9 | 45.5 | 59.3 | 59.1 | — | — | — | |
| DuQuant+LWCModel=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 51.61 | 48.52 | 41.42 | 58.73 | 57.74 | — | — | — | |
| MagRSize=13B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 51.2 | 48.5 | 42.2 | 58.3 | 57.3 | — | — | — | |
| OmniQuantSize=13B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 51 | 47.5 | 42.8 | 58.8 | 56.9 | — | — | — | |
| QAT-LORABackbone Model=LLaMA-1 33B, Precision=3&16-bit2024.02 | 50.6 | — | — | — | — | — | — | — | |
| QLORABackbone Model=LLaMA-2 13B, Precision=3&16-bit2024.02 | 50.6 | — | — | — | — | — | — | — | |
| GPTQBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 50.4 | — | — | — | — | — | — | — | |
| OmniQBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 50.2 | — | — | — | — | — | — | — | |
| QLORABackbone Model=LLaMA-1 33B, Precision=3&16-bit2024.02 | 50.1 | — | — | — | — | — | — | — | |
| L4QBackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 50 | — | — | — | — | — | — | — | |
| L4QModel=LLaMA-2 13B, Precision=3-bit2024.02 | 50 | — | — | — | — | — | — | — | |
| QA-LORABackbone Model=LLaMA-2 13B, Precision=3-bit2024.02 | 49.9 | — | — | — | — | — | — | — | |
| GPTQBackbone Model=Mistral-v0.1 7B, Precision=3-bit2024.02 | 49.6 | — | — | — | — | — | — | — | |
| QAT-LORABackbone Model=LLaMA-2 13B, Precision=3&16-bit2024.02 | 48.9 | — | — | — | — | — | — | — | |
| QA-LORABackbone Model=LLaMA-1 33B, Precision=3-bit2024.02 | 48.7 | — | — | — | — | — | — | — | |
| Original LLaMAModel=LLaMA-13B, Status=Zero-shot baseline2023.09 | 47.4 | — | — | — | — | — | — | — | |
| 4-bit AdamWModel=LLaMA-13B, Status=Fine-tuned2023.09 | 47.4 | — | — | — | — | — | — | — | |
| LORABackbone Model=LLaMA-1 13B, Precision=16-bit2024.02 | 47 | — | — | — | — | — | — | — | |
| LLAMASize=13B, Shots=5-shot2023.05 | 46.9 | — | — | — | — | — | — | — | |
| LLaMA2-7bBit=16, Foundation Model=LLaMA2-7b2024.05 | 46.8 | 51.6 | 37.3 | 52.2 | 49.9 | — | — | — | |
| 32-bit AdamWModel=LLaMA-13B, Status=Fine-tuned2023.09 | 46.5 | — | — | — | — | — | — | — | |
| Pre-trainedBackbone Model=LLaMA-1 13B, Precision=16-bit2024.02 | 46.3 | — | — | — | — | — | — | — | |
| LLM-QFABit=Avg. (2-4 bit), Foundation Model=LLaMA2-13b2024.05 | 46.1 | — | — | — | — | — | — | — | |
| LORABackbone Model=LLaMA-2 7B, Precision=16-bit2024.02 | 46 | — | — | — | — | — | — | — | |
| L4QModel=LLaMA-1 13B, Precision=4-bit2024.02 | 46 | — | — | — | — | — | — | — | |
| FP16Model=LLaMA-2-7B, Quantization=FP162026.04 | 46 | — | — | — | — | — | — | — | |
| fp16Size=7B, Shot count=5-shot, Backbone=LLaMA-22026.02 | 45.9 | 43 | 37.4 | 51.9 | 53.2 | — | — | — | |
| QA-LoRABit=Avg. (2-4 bit), Foundation Model=LLaMA2-13b2024.05 | 45.8 | — | — | — | — | — | — | — | |
| L4QModel=LLaMA-2 7B, Precision=4-bit2024.02 | 45.5 | — | — | — | — | — | — | — | |
| Pre-trainedBackbone Model=LLaMA-2 7B, Precision=16-bit2024.02 | 45.4 | — | — | — | — | — | — | — | |
| COVERCALModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 45.4 | — | — | — | — | — | — | — | |
| GPTQBit=4, Foundation Model=LLaMA2-7b2024.05 | 45.1 | 50.5 | 36.9 | 50.5 | 47.5 | — | — | — | |
| QA-LoRABit=4, Foundation Model=LLaMA2-7b2024.05 | 45.1 | 49.8 | 36.8 | 49.8 | 47.8 | — | — | — | |
| AtomModel=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 45.01 | 39.21 | 35.35 | 59.72 | 45.77 | — | — | — | |
| Max-ActVarModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 44.8 | — | — | — | — | — | — | — | |
| StratifiedModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 44.6 | — | — | — | — | — | — | — | |
| Max-PPLModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 44.5 | — | — | — | — | — | — | — | |
| RandomModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 44.2 | — | — | — | — | — | — | — | |
| LLM-QFABit=4, Foundation Model=LLaMA2-7b2024.05 | 44 | 48.4 | 35.6 | 48.1 | 46.9 | — | — | — |