Language Modeling on C4 (val)
1.7PPLAWQ
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AWQBackbone=Llama 2 7B, Context Length=2048, Quantization=2-bit weight-only PTQ2026.05 | 1.7 | 2.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Backbone=MISTRAL-7B, BIT=162026.07 | 5.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPTQWBits=162023.07 | 5.709 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIPWBits=162023.07 | 5.709 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BF16Model=Llama 3 70B, Bits=16.0, Context Length=81922024.06 | 5.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Type=None, Backbone=LLaMA-3-70B, Speedup=1.0x, Context Length=8192, Precision=16-bit2025.11 | 5.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=22026.07 | 5.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 70B, Bits=4.00, Context Length=81922024.06 | 5.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=LLaMA-3-70B, Speedup=1.7x, Context Length=8192, Precision=4-bit2025.11 | 5.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp+Setting=g6n65500, Target Bits=<2.89, Backbone=Llama-2-70B2025.03 | 5.86 | — | — | 2.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=LLaMA-3-70B, Speedup=2.2x, Context Length=8192, Precision=4-bit2025.11 | 5.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIPWBits=42023.07 | 5.869 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPTQWBits=42023.07 | 5.905 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQType=linear, Backbone=LLaMA-3-70B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 5.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Model=Llama 3 70B, Bits=4.00, Context Length=81922024.06 | 5.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUIP#Type=vector, Backbone=LLaMA-3-70B, Speedup=1.9x, Context Length=8192, Precision=4-bit2025.11 | 5.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 70B, Bits=3.00, Context Length=81922024.06 | 5.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-30B#Bits (W-A-KV)=16-16-162023.05 | 6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniQuantSetting=w3, Target Bits=3, Backbone=Llama-2-70B2025.03 | 6.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIPWBits=32023.07 | 6.135 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=MISTRAL-7B, BIT=22026.07 | 6.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Model=Llama 3 70B, Bits=3.00, Context Length=81922024.06 | 6.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=MISTRAL-7B, BIT=22026.07 | 6.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQ#Bits (W-A-KV)=4-16-16, Model=LLaMA-30B2023.05 | 6.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=8-8-8, Model=LLaMA-30B2023.05 | 6.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp+Setting=g8, Target Bits=<2.15, Backbone=Llama-2-70B, Number of centroids (n)=655002025.03 | 6.44 | — | — | 2.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DenseSparsity=0%, Backbone=Llama-2-13B2025.05 | 6.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DenseSparsity=0%, Latency=-, Model=Llama-2-13B2025.05 | 6.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fp16Bits=W16A16, Weight outlier=false, Activation outlier=false, SNN conversion=false, Timestep=N/A2026.02 | 6.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-13B, Group size=g128, Avg. bit=162024.04 | 6.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-13B, Group size=g64, Avg. bit=162024.04 | 6.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=12026.07 | 6.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Type=None, Backbone=Qwen3-14B, Speedup=1.0x, Context Length=8192, Precision=16-bit2025.11 | 6.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=Qwen3-14B, Speedup=1.7x, Context Length=8192, Precision=4-bit2025.11 | 6.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=Qwen3-14B, Speedup=2.2x, Context Length=8192, Precision=4-bit2025.11 | 6.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KirinBits=W4A(4&8), Weight outlier=true, Activation outlier=true, SNN conversion=true, Timestep=T=162026.02 | 6.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Type=None, Backbone=LLaMA-2-7B, Speedup=1.0x, Context Length=4096, Precision=16-bit2025.11 | 6.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=Qwen3-14B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 6.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Backbone=LLAMA-2-7B, BIT=162026.07 | 6.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpikeQuantBits=W4A(4&5), Weight outlier=false, Activation outlier=true, SNN conversion=true, Timestep=T=322026.02 | 6.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQType=linear, Backbone=Qwen3-14B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 6.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=LLaMA-2-7B, Speedup=1.7x, Context Length=4096, Precision=4-bit2025.11 | 6.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-13B#Bits (W-A-KV)=16-16-162023.05 | 6.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=LLaMA-3-70B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 6.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=LLaMA-2-7B, Speedup=2.2x, Context Length=4096, Precision=4-bit2025.11 | 6.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUIP#Type=vector, Backbone=LLaMA-2-7B, Speedup=1.9x, Context Length=4096, Precision=4-bit2025.11 | 6.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CherryQModel=LLaMA2-13B, Group size=g64, Avg. bit=3.292024.04 | 6.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=LLaMA-2-7B, Speedup=2.4x, Context Length=4096, Precision=4-bit2025.11 | 6.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 70B, Bits=2.00, Context Length=81922024.06 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-8-8, Model=LLaMA-13B2023.05 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQ#Bits (W-A-KV)=4-16-16, Model=LLaMA-13B2023.05 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CherryQModel=LLaMA2-13B, Group size=g128, Avg. bit=3.172024.04 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQType=linear, Backbone=LLaMA-2-7B, Speedup=2.4x, Context Length=4096, Precision=4-bit2025.11 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMNIQType=linear, Backbone=LLaMA-2-7B, Speedup=2.4x, Context Length=4096, Precision=4-bit2025.11 | 6.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SqueezeLLMModel=LLaMA2-13B, Group size=g64, Avg. bit=3.242024.04 | 6.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPINQType=linear, Backbone=LLaMA-2-7B, Speedup=2.4x, Context Length=4096, Precision=4-bit2025.11 | 6.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-2-7B, BIT=22026.07 | 6.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-8-8, Model=LLaMA-30B2023.05 | 6.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQModel=LLaMA2-13B, Group size=g128, Avg. bit=3.152024.04 | 6.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQSetting=w4g128, Target Bits=4, Backbone=Llama-3-70B2025.03 | 6.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-7B, Group size=g128, Avg. bit=162024.04 | 6.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-7B, Group size=g64, Avg. bit=162024.04 | 6.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Type=None, Backbone=Qwen3-8B, Speedup=1.0x, Context Length=8192, Precision=16-bit2025.11 | 6.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=–, Backbone=LLAMA-2-7B2026.04 | 6.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniQuantModel=LLaMA2-13B, Group size=g128, Avg. bit=3.152024.04 | 6.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp+Setting=g4n65500, Target Bits=<4.13, Backbone=Llama-3-70B2025.03 | 6.99 | — | — | 4.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=8-8-8, Model=LLaMA-7B2023.05 | 7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=8-8-8, Model=LLaMA-13B2023.05 | 7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QATModel=LLaMA2-13B, Group size=g64, Avg. bit=3.252024.04 | 7.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp+Setting=g9, Target Bits=<2.01, Backbone=Llama-2-70B, Number of centroids (n)=655002025.03 | 7.02 | — | — | 1.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=Qwen3-8B, Speedup=1.7x, Context Length=8192, Precision=4-bit2025.11 | 7.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AtomBits=W4A4, Weight outlier=false, Activation outlier=true, SNN conversion=false, Timestep=N/A2026.02 | 7.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=Qwen3-8B, Speedup=2.2x, Context Length=8192, Precision=4-bit2025.11 | 7.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| YAQA-BBit-width=INT4, Backbone=Llama 2 7B, Finetuning=False2025.05 | 7.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CBQBit-width=INT4, Backbone=Llama 2 7B, Finetuning=False2025.05 | 7.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fp16Bits=W16A16, Weight outlier=false, Activation outlier=false, SNN conversion=false, Timestep=N/A2026.02 | 7.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=Qwen3-8B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 7.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPTQWBits=32023.07 | 7.099 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BF16Model=Llama 3 8B, Bits=16.0, Context Length=81922024.06 | 7.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQModel=LLaMA2-13B, Group size=g64, Avg. bit=3.32024.04 | 7.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Type=None, Backbone=LLaMA-3-8B, Speedup=1.0x, Context Length=8192, Precision=16-bit2025.11 | 7.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Backbone=LLAMA-3-8B, BIT=162026.07 | 7.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQType=linear, Backbone=Qwen3-8B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 7.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp+Setting=g4n65500, Target Bits=<4.13, Backbone=Llama-1-7B2025.03 | 7.17 | — | — | 4.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QATModel=LLaMA2-13B, Group size=g128, Avg. bit=3.132024.04 | 7.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 8B, Bits=4.00, Context Length=81922024.06 | 7.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-7B#Bits (W-A-KV)=16-16-162023.05 | 7.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQSetting=w4g128, Target Bits=4, Backbone=Llama-1-7B2025.03 | 7.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=LLaMA-3-8B, Speedup=1.7x, Context Length=8192, Precision=4-bit2025.11 | 7.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-2-7B, BIT=22026.07 | 7.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTQModel=LLaMA2-13B, Group size=g128, Avg. bit=3.152024.04 | 7.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KirinBits=W4A(4&8), Weight outlier=true, Activation outlier=true, SNN conversion=true, Timestep=T=162026.02 | 7.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp-Setting=g4n65500, Target Bits=<4.14, Backbone=Llama-1-7B2025.03 | 7.27 | — | — | 4.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=LLaMA-3-8B, Speedup=2.2x, Context Length=8192, Precision=4-bit2025.11 | 7.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-8-4, Model=LLaMA-30B2023.05 | 7.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Model=Llama 3 8B, Bits=4.00, Context Length=81922024.06 | 7.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUIP#Type=vector, Backbone=LLaMA-3-8B, Speedup=1.9x, Context Length=8192, Precision=4-bit2025.11 | 7.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpikeQuantBits=W4A(4&5), Weight outlier=false, Activation outlier=true, SNN conversion=true, Timestep=T=322026.02 | 7.33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CherryQModel=LLaMA2-7B, Group size=g64, Avg. bit=3.32024.04 | 7.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=LLaMA-3-8B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 7.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |