Language Modeling on C4 (Loss)
2.55C4 LossOriginal Model (no compression)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Original Model (no compression)Model=Qwen2-7B2026.01 | 2.55 | — | |
| AQLMModel=Qwen2-7B, Compression level=High, Bits per parameter (BPP)=3.01, Training budget=50 million tokens2026.01 | 2.74 | — | |
| VQ-then-trainModel=Qwen2-7B, Compression level=High, Bits per parameter (BPP)=3.01, Training budget=50 million tokens2026.01 | 2.81 | — | |
| VQ-then-trainModel=Qwen2-7B, Compression level=Medium, Bits per parameter (BPP)=2.26, Training budget=50 million tokens2026.01 | 3.16 | — | |
| VQ-then-trainModel=Qwen2-7B, Compression level=Low, Bits per parameter (BPP)=1.5, Training budget=50 million tokens2026.01 | 3.22 | — | |
| Original Model (no compression)Model=Qwen2-1.5B2026.01 | 3.28 | — | |
| AQLMModel=Qwen2-7B, Compression level=Medium, Bits per parameter (BPP)=2.26, Training budget=50 million tokens2026.01 | 3.28 | — | |
| VQ-then-trainModel=Qwen2-1.5B, Compression level=High, Bits per parameter (BPP)=2.03, Training budget=1 billion tokens2026.01 | 3.33 | — | |
| AQLMModel=Qwen2-7B, Compression level=Low, Bits per parameter (BPP)=1.5, Training budget=50 million tokens2026.01 | 3.37 | — | |
| AQLMModel=Qwen2-1.5B, Compression level=High, Bits per parameter (BPP)=2.03, Training budget=1 billion tokens2026.01 | 3.65 | — | |
| VQ-then-trainModel=Qwen2-1.5B, Compression level=Medium, Bits per parameter (BPP)=1.15, Training budget=1 billion tokens2026.01 | 3.69 | — | |
| VQ-then-trainModel=Qwen2-1.5B, Compression level=Low, Bits per parameter (BPP)=0.58, Training budget=1 billion tokens2026.01 | 4.27 | — | |
| FP16Model=Llama-13B, #Bits=W16A162026.03 | 5.62 | — | |
| Random WeightsModel=Qwen2-7B2026.01 | 6 | — | |
| Random WeightsModel=Qwen2-1.5B2026.01 | 6.07 | — | |
| AQLMModel=Qwen2-1.5B, Compression level=Medium, Bits per parameter (BPP)=1.15, Training budget=1 billion tokens2026.01 | 6.39 | — | |
| FP16Model=L2-13B, Bits=16, Group size=1282026.02 | 6.47 | — | |
| GPTAQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=1019.22026.02 | 6.57 | — | |
| SNRQ-CModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=726.12026.02 | 6.57 | — | |
| SNRQ-SModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=709.42026.02 | 6.57 | — | |
| LDLQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=623.92026.02 | 6.58 | — | |
| GPTQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=727.72026.02 | 6.59 | — | |
| SpQRModel=Llama-13B, #Bits=W3.45A162026.03 | 6.72 | — | |
| SliderQuantModel=Llama-13B, #Bits=W3A162026.03 | 6.78 | — | |
| FP16Model=L2-7B, Bits=16, Group size=1282026.02 | 6.97 | — | |
| FP16Format=FP16, Quantization=None2025.11 | 6.97 | — | |
| SNRQ-S+RHTModel=L2-13B, Quantization Time (s)=970.22026.02 | 7.03 | — | |
| SNRQ-C+RHTModel=L2-13B, Quantization Time (s)=1106.32026.02 | 7.05 | — | |
| SNRQ-SModel=L2-13B, Quantization Time (s)=786.42026.02 | 7.07 | — | |
| GPTAQ+RHTModel=L2-13B, Quantization Time (s)=1259.42026.02 | 7.07 | — | |
| SNRQ-SModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=724.22026.02 | 7.07 | — | |
| FP16Model=Llama-7B, #Bits=W16A162026.03 | 7.08 | — | |
| GPTAQModel=L2-13B, Quantization Time (s)=1012.02026.02 | 7.1 | — | |
| SNRQ-CModel=L2-13B, Quantization Time (s)=965.02026.02 | 7.1 | — | |
| GPTAQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=10122026.02 | 7.1 | — | |
| SNRQ-CModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=749.62026.02 | 7.1 | — | |
| SNRQ-CModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=450.92026.02 | 7.12 | — | |
| SNRQ-SModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=405.62026.02 | 7.12 | — | |
| LDLQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=719.22026.02 | 7.12 | — | |
| SliderQuantModel=Llama-7B, #Bits=W3A162026.03 | 7.13 | — | |
| GPTAQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=610.52026.02 | 7.14 | — | |
| QuIPModel=L2-13B, Quantization Time (s)=1096.22026.02 | 7.15 | — | |
| GPTQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=481.62026.02 | 7.16 | — | |
| LDLQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=404.82026.02 | 7.16 | — | |
| GPTQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=820.52026.02 | 7.16 | — | |
| SpQRModel=Llama-7B, #Bits=W3.45A162026.03 | 7.28 | — | |
| ParoQuantFormat=MXFP4, Quantization=W4A4, Weight Quantization=GPTQ2025.11 | 7.44 | — | |
| MR-GPTQFormat=MXFP4, Quantization=W4A4, Weight Quantization=GPTQ2025.11 | 7.71 | — | |
| SNRQ-S+RHTModel=L2-7B, Quantization Time (s)=644.52026.02 | 7.77 | — | |
| SNRQ-C+RHTModel=L2-7B, Quantization Time (s)=697.32026.02 | 7.82 | — | |
| GPTAQ+RHTModel=L2-7B, Quantization Time (s)=795.72026.02 | 7.86 | — | |
| SNRQ-SModel=L2-7B, Quantization Time (s)=449.02026.02 | 7.88 | — | |
| SNRQ-SModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=417.32026.02 | 7.88 | — | |
| SNRQ-CModel=L2-7B, Quantization Time (s)=517.12026.02 | 7.89 | — | |
| SNRQ-CModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=407.42026.02 | 7.89 | — | |
| QuIPModel=L2-7B, Quantization Time (s)=704.82026.02 | 8.02 | — | |
| GPTAQModel=L2-7B, Quantization Time (s)=590.42026.02 | 8.04 | — | |
| GPTAQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=590.42026.02 | 8.04 | — | |
| LDLQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=427.82026.02 | 8.12 | — | |
| GPTQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=467.92026.02 | 8.26 | — | |
| AQLMModel=Qwen2-1.5B, Compression level=Low, Bits per parameter (BPP)=0.58, Training budget=1 billion tokens2026.01 | 8.49 | — | |
| FP16Model=L3-8B, Bits=16, Group size=1282026.02 | 8.93 | — | |
| Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 9.05 | — | |
| Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.03 | 9.05 | — | |
| Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=60%2026.03 | 9.05 | — | |
| OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 9.38 | — | |
| SNRQ-CModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=495.22026.02 | 9.44 | — | |
| SNRQ-SModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=498.72026.02 | 9.44 | — | |
| GPTAQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=703.22026.02 | 9.49 | — | |
| LDLQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=425.12026.02 | 9.52 | — | |
| GPTQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=4892026.02 | 9.53 | — | |
| Camera-PModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 9.84 | — | |
| HEAPrModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 9.85 | — | |
| NAEEModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 10.07 | — | |
| OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.03 | 10.75 | — | |
| SNRQ-S+RHTModel=L3-8B, Quantization Time (s)=688.72026.02 | 11.37 | — | |
| SNRQ-C+RHTModel=L3-8B, Quantization Time (s)=741.82026.02 | 11.58 | — | |
| OursModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.03 | 11.58 | — | |
| GPTAQ+RHTModel=L3-8B, Quantization Time (s)=853.12026.02 | 11.61 | — | |
| Camera-PModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.03 | 11.68 | — | |
| SNRQ-SModel=L3-8B, Quantization Time (s)=542.82026.02 | 11.73 | — | |
| SNRQ-SModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=484.72026.02 | 11.73 | — | |
| QuIPModel=L3-8B, Quantization Time (s)=763.82026.02 | 11.8 | — | |
| SNRQ-CModel=L3-8B, Quantization Time (s)=594.42026.02 | 11.87 | — | |
| SNRQ-CModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=492.32026.02 | 11.87 | — | |
| Baseline#Bits=W16A162026.03 | 12.08 | — | |
| OursModel=Qwen3-30B-A3B, Sparsity Ratio=40%2026.03 | 12.1 | — | |
| Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=20%2026.03 | 12.13 | — | |
| Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=40%2026.03 | 12.13 | — | |
| Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=60%2026.03 | 12.13 | — | |
| GPTAQModel=L3-8B, Quantization Time (s)=715.02026.02 | 12.15 | — | |
| GPTAQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=7152026.02 | 12.15 | — | |
| Camera-PModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.03 | 12.25 | — | |
| HEAPrModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.03 | 12.34 | — | |
| NAEEModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.03 | 12.44 | — | |
| SliderQuant#Bits=W4A162026.03 | 12.47 | — | |
| OmniQuant#Bits=W4A162026.03 | 12.54 | — | |
| D2-MoEModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.03 | 12.62 | — | |
| OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=60%2026.03 | 12.65 | — | |
| GPTQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=547.52026.02 | 12.74 | — |