Perplexity on C4
6.24PerplexityFP16 Baseline
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FP16 BaselineAvg. Bitwidth=16.000, Model=Llama-2-7B2025.07 | 6.24 | — | |
| float16Backbone=Llama 3.1/3.2, Scale=70B, Precision=float162024.12 | 6.68 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W8A8, 8-bit percentage=100%2024.12 | 6.68 | — | |
| HPTQAvg. Bitwidth=4.125, Model=Llama-2-7B2025.07 | 6.73 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W6A8, 8-bit percentage=50%2024.12 | 6.73 | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=70B, Precision=W8A82024.12 | 6.77 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 6.79 | — | |
| SSQR-1%Avg. Bitwidth=4.445, Model=Llama-2-7B2025.07 | 6.81 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 6.83 | — | |
| GPTQAvg. Bitwidth=4.125, Model=Llama-2-7B2025.07 | 6.9 | — | |
| HPTQAvg. Bitwidth=3.125, Model=Llama-2-7B2025.07 | 7.04 | — | |
| QServeBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A82024.12 | 7.07 | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A82024.12 | 7.1 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A8, 8-bit percentage=0%2024.12 | 7.24 | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A162024.12 | 7.43 | — | |
| SSQR-1%Avg. Bitwidth=3.445, Model=Llama-2-7B2025.07 | 7.52 | — | |
| float16Backbone=Mistral, Scale=7B v0.3, Precision=float162024.12 | 7.84 | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W8A8, 8-bit percentage=100%2024.12 | 7.84 | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W6A8, 8-bit percentage=50%2024.12 | 7.87 | — | |
| SmoothQuantBackbone=Mistral, Scale=7B v0.3, Precision=W8A82024.12 | 7.88 | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 7.9 | — | |
| Basic RTNBackbone=Mistral, Scale=7B v0.3, Precision=W5A162024.12 | 7.91 | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 7.93 | — | |
| QServeBackbone=Mistral, Scale=7B v0.3, Precision=W4A82024.12 | 7.98 | — | |
| QuaRotBackbone=Mistral, Scale=7B v0.3, Precision=W4A82024.12 | 7.99 | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4A8, 8-bit percentage=0%2024.12 | 8.02 | — | |
| Basic RTNBackbone=Mistral, Scale=7B v0.3, Precision=W4A162024.12 | 8.04 | — | |
| GPTQAvg. Bitwidth=3.125, Model=Llama-2-7B2025.07 | 8.08 | — | |
| QuaRotBackbone=Mistral, Scale=7B v0.3, Precision=W4A42024.12 | 8.5 | — | |
| float16Backbone=Llama 3.1/3.2, Scale=8B, Precision=float162024.12 | 8.95 | — | |
| float16Backbone=Qwen2.5, Scale=32B, Precision=float162024.12 | 8.95 | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W8A8, 8-bit percentage=100%2024.12 | 8.96 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W8A8, 8-bit percentage=100%2024.12 | 8.97 | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W6A8, 8-bit percentage=50%2024.12 | 8.98 | — | |
| Basic RTNBackbone=Qwen2.5, Scale=32B, Precision=W5A162024.12 | 8.99 | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 9 | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 9.01 | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=32B, Precision=W8A82024.12 | 9.04 | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=8B, Precision=W8A82024.12 | 9.08 | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4A8, 8-bit percentage=0%2024.12 | 9.08 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W6A8, 8-bit percentage=50%2024.12 | 9.09 | — | |
| QuaRotBackbone=Qwen2.5, Scale=32B, Precision=W4A82024.12 | 9.1 | — | |
| Basic RTNBackbone=Qwen2.5, Scale=32B, Precision=W4A162024.12 | 9.14 | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=8B, Precision=W5A162024.12 | 9.15 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 9.22 | — | |
| QServeBackbone=Qwen2.5, Scale=32B, Precision=W4A82024.12 | 9.31 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 9.33 | — | |
| HPTQAvg. Bitwidth=2.125, Model=Llama-2-7B2025.07 | 9.43 | — | |
| QServeBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A82024.12 | 9.49 | — | |
| BF16Backbone=Llama-3.1-8B, Weight precision=BF16, Activation precision=BF162026.05 | 9.5 | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=70B, Precision=W5A162024.12 | 9.52 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A8, 8-bit percentage=0%2024.12 | 9.62 | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A82024.12 | 9.67 | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A162024.12 | 9.72 | — | |
| AAACBackbone=Llama-3.1-8B, Weight precision=W4, Activation precision=A16, Group size=16, S parameter=162026.05 | 9.84 | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A42024.12 | 9.91 | — | |
| QuaRotBackbone=Qwen2.5, Scale=32B, Precision=W4A42024.12 | 9.98 | — | |
| IF4Backbone=Llama-3.1-8B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.07 | — | |
| AWQBackbone=Llama-3.1-8B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.07 | — | |
| RTNBackbone=Llama-3.1-8B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.21 | — | |
| float16Backbone=Qwen2.5, Scale=7B, Precision=float162024.12 | 10.44 | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W8A8, 8-bit percentage=100%2024.12 | 10.45 | — | |
| BF16Backbone=Qwen2.5-27B, Weight precision=BF16, Activation precision=BF162026.05 | 10.47 | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W6A8, 8-bit percentage=50%2024.12 | 10.49 | — | |
| Basic RTNBackbone=Qwen2.5, Scale=7B, Precision=W5A162024.12 | 10.53 | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 10.53 | — | |
| AAACBackbone=Qwen2.5-27B, Weight precision=W4, Activation precision=A16, Group size=16, S parameter=162026.05 | 10.55 | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 10.57 | — | |
| AWQBackbone=Qwen2.5-27B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.59 | — | |
| IF4Backbone=Qwen2.5-27B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.61 | — | |
| RTNBackbone=Qwen2.5-27B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 10.64 | — | |
| QuaRotBackbone=Qwen2.5, Scale=7B, Precision=W4A82024.12 | 10.68 | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4A8, 8-bit percentage=0%2024.12 | 10.75 | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=7B, Precision=W8A82024.12 | 10.85 | — | |
| Basic RTNBackbone=Qwen2.5, Scale=7B, Precision=W4A162024.12 | 10.88 | — | |
| QServeBackbone=Qwen2.5, Scale=7B, Precision=W4A82024.12 | 11.06 | — | |
| BF16Backbone=Llama-3.2-3B, Weight precision=BF16, Activation precision=BF162026.05 | 11.2 | — | |
| AAACBackbone=Llama-3.2-3B, Weight precision=W4, Activation precision=A16, Group size=16, S parameter=162026.05 | 11.58 | — | |
| AWQBackbone=Llama-3.2-3B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 11.81 | — | |
| IF4Backbone=Llama-3.2-3B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 11.86 | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A42024.12 | 11.95 | — | |
| RTNBackbone=Llama-3.2-3B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 12.02 | — | |
| QuaRotBackbone=Qwen2.5, Scale=7B, Precision=W4A42024.12 | 12.05 | — | |
| BF16Backbone=Qwen2.5-9B, Weight precision=BF16, Activation precision=BF162026.05 | 12.19 | — | |
| AAACBackbone=Qwen2.5-9B, Weight precision=W4, Activation precision=A16, Group size=16, S parameter=162026.05 | 12.38 | — | |
| AWQBackbone=Qwen2.5-9B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 12.44 | — | |
| IF4Backbone=Qwen2.5-9B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 12.53 | — | |
| RTNBackbone=Qwen2.5-9B, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 12.57 | — | |
| BF16Backbone=Average, Weight precision=BF16, Activation precision=BF162026.05 | 12.67 | — | |
| float16Backbone=Llama 3.1/3.2, Scale=1B, Precision=float162024.12 | 12.72 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W8A8, 8-bit percentage=100%2024.12 | 12.75 | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=1B, Precision=W8A82024.12 | 12.91 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W6A8, 8-bit percentage=50%2024.12 | 12.98 | — | |
| AAACBackbone=Average, Weight precision=W4, Activation precision=A16, Group size=16, S parameter=162026.05 | 13.05 | 50.9 | |
| float16Backbone=Qwen2.5, Scale=1.5B, Precision=float162024.12 | 13.11 | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W8A8, 8-bit percentage=100%2024.12 | 13.14 | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=1B, Precision=W5A162024.12 | 13.25 | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 13.25 | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W6A8, 8-bit percentage=50%2024.12 | 13.25 | — | |
| AWQBackbone=Average, Weight precision=W4, Activation precision=A16, Group size=162026.05 | 13.25 | 26 |