Perplexity on WikiText-2
2.81Perplexityfloat16
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| float16Backbone=Llama 3.1/3.2, Scale=70B, Precision=float162024.12 | 2.81 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W8A8, 8-bit percentage=100%2024.12 | 2.81 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W6A8, 8-bit percentage=50%2024.12 | 2.86 | — | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=70B, Precision=W8A82024.12 | 2.92 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 2.97 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 3.02 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=70B, Precision=W5A162024.12 | 3.16 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A8, 8-bit percentage=0%2024.12 | 3.3 | — | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A82024.12 | 3.43 | — | — | |
| QServeBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A82024.12 | 3.49 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A162024.12 | 3.55 | — | — | |
| float16Backbone=Qwen2.5, Scale=32B, Precision=float162024.12 | 5.02 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W8A8, 8-bit percentage=100%2024.12 | 5.02 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W6A8, 8-bit percentage=50%2024.12 | 5.05 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=32B, Precision=W5A162024.12 | 5.09 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 5.09 | — | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=32B, Precision=W8A82024.12 | 5.12 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 5.12 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=32B, Precision=W4A8, 8-bit percentage=0%2024.12 | 5.21 | — | — | |
| QuaRotBackbone=Qwen2.5, Scale=32B, Precision=W4A82024.12 | 5.23 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=32B, Precision=W4A162024.12 | 5.27 | — | — | |
| float16Backbone=Mistral, Scale=7B v0.3, Precision=float162024.12 | 5.32 | — | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W8A8, 8-bit percentage=100%2024.12 | 5.32 | — | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W6A8, 8-bit percentage=50%2024.12 | 5.33 | — | — | |
| SmoothQuantBackbone=Mistral, Scale=7B v0.3, Precision=W8A82024.12 | 5.35 | — | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 5.35 | — | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 5.36 | — | — | |
| Basic RTNBackbone=Mistral, Scale=7B v0.3, Precision=W5A162024.12 | 5.38 | — | — | |
| QuaRotBackbone=Mistral, Scale=7B v0.3, Precision=W4A82024.12 | 5.4 | — | — | |
| MixLLMBackbone=Mistral, Scale=7B v0.3, Precision=W4A8, 8-bit percentage=0%2024.12 | 5.42 | — | — | |
| QServeBackbone=Mistral, Scale=7B v0.3, Precision=W4A82024.12 | 5.44 | — | — | |
| FP16 BaselineAvg. Bitwidth=16.000, Model=Llama-2-7B2025.07 | 5.5 | — | — | |
| Basic RTNBackbone=Mistral, Scale=7B v0.3, Precision=W4A162024.12 | 5.51 | — | — | |
| HPTQAvg. Bitwidth=4.125, Model=Llama-2-7B2025.07 | 5.53 | — | — | |
| QServeBackbone=Qwen2.5, Scale=32B, Precision=W4A82024.12 | 5.55 | — | — | |
| SSQR-1%Avg. Bitwidth=4.445, Model=Llama-2-7B2025.07 | 5.6 | — | — | |
| GPTQAvg. Bitwidth=4.125, Model=Llama-2-7B2025.07 | 5.7 | — | — | |
| HPTQAvg. Bitwidth=3.125, Model=Llama-2-7B2025.07 | 5.77 | — | — | |
| QuaRotBackbone=Mistral, Scale=7B v0.3, Precision=W4A42024.12 | 5.83 | — | — | |
| SSQR-1%Avg. Bitwidth=3.445, Model=Llama-2-7B2025.07 | 6.09 | — | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=70B, Precision=W4A42024.12 | 6.16 | — | — | |
| float16Backbone=Llama 3.1/3.2, Scale=8B, Precision=float162024.12 | 6.24 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W8A8, 8-bit percentage=100%2024.12 | 6.25 | — | — | |
| QuaRotBackbone=Qwen2.5, Scale=32B, Precision=W4A42024.12 | 6.26 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W6A8, 8-bit percentage=50%2024.12 | 6.3 | — | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=8B, Precision=W8A82024.12 | 6.34 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 6.37 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=8B, Precision=W5A162024.12 | 6.4 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 6.42 | — | — | |
| FP16 FA-2T (Context Length)=4K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 6.48 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A8, 8-bit percentage=0%2024.12 | 6.54 | — | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A82024.12 | 6.6 | — | — | |
| QServeBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A82024.12 | 6.64 | — | — | |
| Block-GTQT (Context Length)=4K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 6.66 | 2.8 | 53 | |
| GPTQAvg. Bitwidth=3.125, Model=Llama-2-7B2025.07 | 6.75 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A162024.12 | 6.82 | — | — | |
| float16Backbone=Qwen2.5, Scale=7B, Precision=float162024.12 | 6.85 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W8A8, 8-bit percentage=100%2024.12 | 6.86 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W6A8, 8-bit percentage=50%2024.12 | 6.88 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 6.9 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 6.92 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=7B, Precision=W5A162024.12 | 6.95 | — | — | |
| QuaRotBackbone=Qwen2.5, Scale=7B, Precision=W4A82024.12 | 7.03 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=7B, Precision=W4A8, 8-bit percentage=0%2024.12 | 7.03 | — | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=7B, Precision=W8A82024.12 | 7.17 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=7B, Precision=W4A162024.12 | 7.23 | — | — | |
| QServeBackbone=Qwen2.5, Scale=7B, Precision=W4A82024.12 | 7.39 | — | — | |
| HPTQAvg. Bitwidth=2.125, Model=Llama-2-7B2025.07 | 7.45 | — | — | |
| FP16 FA-2T (Context Length)=64K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 7.47 | — | — | |
| Block-GTQT (Context Length)=64K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 7.72 | 3.3 | 1,486 | |
| FP16 FA-2T (Context Length)=16K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 7.84 | — | — | |
| Block-GTQT (Context Length)=16K, Model=Qwen2.5-3B-Instruct, Deployment Setting=K3V32026.06 | 8.12 | 3.6 | 36,879 | |
| QuaRotBackbone=Qwen2.5, Scale=7B, Precision=W4A42024.12 | 8.15 | — | — | |
| QuaRotBackbone=Llama 3.1/3.2, Scale=8B, Precision=W4A42024.12 | 8.34 | — | — | |
| float16Backbone=Qwen2.5, Scale=1.5B, Precision=float162024.12 | 9.26 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W8A8, 8-bit percentage=100%2024.12 | 9.28 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W6A8, 8-bit percentage=50%2024.12 | 9.33 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 9.4 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 9.44 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=1.5B, Precision=W5A162024.12 | 9.52 | — | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=1.5B, Precision=W8A82024.12 | 9.63 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=1.5B, Precision=W4A8, 8-bit percentage=0%2024.12 | 9.66 | — | — | |
| float16Backbone=Llama 3.1/3.2, Scale=1B, Precision=float162024.12 | 9.75 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W8A8, 8-bit percentage=100%2024.12 | 9.76 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W6A8, 8-bit percentage=50%2024.12 | 9.85 | — | — | |
| SmoothQuantBackbone=Llama 3.1/3.2, Scale=1B, Precision=W8A82024.12 | 9.89 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 9.95 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 10.05 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=1B, Precision=W5A162024.12 | 10.15 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=1.5B, Precision=W4A162024.12 | 10.35 | — | — | |
| MixLLMBackbone=Llama 3.1/3.2, Scale=1B, Precision=W4A8, 8-bit percentage=0%2024.12 | 10.36 | — | — | |
| Basic RTNBackbone=Llama 3.1/3.2, Scale=1B, Precision=W4A162024.12 | 11.72 | — | — | |
| float16Backbone=Qwen2.5, Scale=0.5B, Precision=float162024.12 | 13.07 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=0.5B, Precision=W8A8, 8-bit percentage=100%2024.12 | 13.12 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=0.5B, Precision=W6A8, 8-bit percentage=50%2024.12 | 13.21 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=0.5B, Precision=W4.8A8, 8-bit percentage=20%2024.12 | 13.32 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=0.5B, Precision=W4.4A8, 8-bit percentage=10%2024.12 | 13.42 | — | — | |
| Basic RTNBackbone=Qwen2.5, Scale=0.5B, Precision=W5A162024.12 | 13.61 | — | — | |
| SmoothQuantBackbone=Qwen2.5, Scale=0.5B, Precision=W8A82024.12 | 13.84 | — | — | |
| MixLLMBackbone=Qwen2.5, Scale=0.5B, Precision=W4A8, 8-bit percentage=0%2024.12 | 14.43 | — | — |