Language Modeling on WikiText2 (val)
3.03Perplexity (PPL)TCN-SEQ-J
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| TCN-SEQ-JBackbone=GP-VAE2025.12 | 3.03 | — | — | — | |
| TCN-SEQ-KBackbone=GP-VAE2025.12 | 3.05 | — | — | — | |
| DenseRatio=0%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 3.12 | — | — | — | |
| TCN-SEQ-BBackbone=GP-VAE2025.12 | 3.27 | — | — | — | |
| TCN-SEQ-CBackbone=GP-VAE2025.12 | 3.34 | — | — | — | |
| TCN-SEQ-IBackbone=GP-VAE2025.12 | 3.35 | — | — | — | |
| SoLARatio=20%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 4.06 | — | — | — | |
| SoLARatio=30%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 4.44 | — | — | — | |
| DenseSparsity=0%, Backbone=Llama-2-13B2025.05 | 4.57 | — | — | — | |
| DenseSparsity=0%, Latency=-, Model=Llama-2-13B2025.05 | 4.57 | — | — | — | |
| DenseRatio=0%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 4.57 | — | — | — | |
| DenseRatio=0%, Backbone=Mistral, Model Scale=7B2026.03 | 4.92 | — | — | — | |
| DenseRatio=0%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 5.11 | — | — | — | |
| EvoPressSparsity=12.5%, Latency=29m, Model=Llama-2-13B2025.05 | 5.42 | — | — | — | |
| EvoPressSparsity=50%, Latency=139m, Backbone=Llama-2-13B2025.05 | 5.45 | — | — | — | |
| FP16Backbone=Llama-2-7B, Quantizable linear layers=2242026.03 | 5.51 | 13.5 | 267 | — | |
| UniCuCoSparsity=50%, Latency=<1s, Backbone=Llama-2-13B2025.05 | 5.53 | — | — | — | |
| UniformSparsity=50%, Latency=<1s, Backbone=Llama-2-13B2025.05 | 5.54 | — | — | — | |
| RAMPBackbone=Llama-2-7B, Quantizable linear layers=2242026.03 | 5.54 | 3.68 | — | — | |
| AWQ-4Backbone=Llama-2-7B, Quantizable linear layers=224, Quantization Bit-width=42026.03 | 5.6 | 3.9 | 6 | 1.1 | |
| OWLSparsity=50%, Latency=114m, Backbone=Llama-2-13B2025.05 | 5.61 | — | — | — | |
| Q4_K_MBackbone=Llama-2-7B, Quantizable linear layers=224, Quantization Bit-width=42026.03 | 5.61 | 3.95 | 7.3 | 1.3 | |
| SoLARatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 5.61 | — | — | — | |
| QUIP#-4Backbone=Llama-2-7B, Quantizable linear layers=224, Quantization Bit-width=42026.03 | 5.62 | 3.9 | 6 | 1.4 | |
| Cosine (Window)Sparsity=12.5%, Latency=16s, Model=Llama-2-13B2025.05 | 5.67 | — | — | — | |
| GPTQ-4Backbone=Llama-2-7B, Quantizable linear layers=224, Quantization Bit-width=42026.03 | 5.69 | 3.9 | 6 | 2.7 | |
| SliceGPTRatio=20%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 5.76 | — | — | — | |
| ShortGPTSparsity=12.5%, Latency=<1s, Model=Llama-2-13B2025.05 | 5.88 | — | — | — | |
| FLAPRatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 5.9 | — | — | — | |
| UniCuCoSparsity=12.5%, Latency=<1s, Model=Llama-2-13B2025.05 | 5.94 | — | — | — | |
| RTN-4Backbone=Llama-2-7B, Quantizable linear layers=224, Quantization Bit-width=42026.03 | 5.94 | 3.9 | 6 | 7.2 | |
| SVD-LLMRatio=20%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 5.96 | — | — | — | |
| Weight SubcloningSparsity=12.5%, Latency=<1s, Model=Llama-2-13B2025.05 | 5.97 | — | — | — | |
| SoLARatio=20%, Backbone=Mistral, Model Scale=7B2026.03 | 6.06 | — | — | — | |
| SVD-LLMRatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 6.18 | — | — | — | |
| SoLARatio=30%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 6.31 | — | — | — | |
| BolacoRatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 6.34 | — | — | — | |
| FP16Model Architecture=Llama-3-8B2025.12 | 6.41 | — | — | — | |
| SoLARatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 6.52 | — | — | — | |
| FLAPRatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 6.76 | — | — | — | |
| CALMModel Architecture=Llama-3-8B2025.12 | 6.89 | — | — | — | |
| SVD-LLMRatio=30%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 6.95 | — | — | — | |
| LLaMA-30B#Bits (W-A-KV)=16-16-162023.05 | 7 | — | — | — | |
| NeUQIModel=LLaMA 2, Size=70B, Bits=22025.05 | 7.03 | — | — | — | |
| FLAPRatio=30%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 7.08 | — | — | — | |
| FLAPRatio=20%, Backbone=Mistral, Model Scale=7B2026.03 | 7.11 | — | — | — | |
| EvoPressSparsity=60%, Latency=138m, Backbone=Llama-2-13B2025.05 | 7.15 | — | — | — | |
| EvoPressSparsity=25%, Latency=28.6m, Model=Llama-2-13B2025.05 | 7.2 | — | — | — | |
| SVD-LLMRatio=20%, Backbone=Mistral, Model Scale=7B2026.03 | 7.26 | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-16-16, Model=LLaMA-30B2023.05 | 7.3 | — | — | — | |
| BolacoRatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 7.31 | — | — | — | |
| SmoothQuantModel Architecture=Llama-3-8B2025.12 | 7.32 | — | — | — | |
| OWLSparsity=60%, Latency=98m, Backbone=Llama-2-13B2025.05 | 7.33 | — | — | — | |
| SoLARatio=30%, Backbone=Mistral, Model Scale=7B2026.03 | 7.38 | — | — | — | |
| SpinQuantModel Architecture=Llama-3-8B2025.12 | 7.48 | — | — | — | |
| UniCuCoSparsity=60%, Latency=<1s, Backbone=Llama-2-13B2025.05 | 7.6 | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-8-4, Model=LLaMA-30B2023.05 | 7.7 | — | — | — | |
| SoLARatio=30%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 7.81 | — | — | — | |
| GPTQ#Bits (W-A-KV)=4-16-16, Model=LLaMA-30B2023.05 | 7.9 | — | — | — | |
| SVD-LLMRatio=30%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 7.93 | — | — | — | |
| SVD-LLMRatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 8.07 | — | — | — | |
| SliceGPTRatio=30%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 8.09 | — | — | — | |
| AWQModel Architecture=Llama-3-8B2025.12 | 8.12 | — | — | — | |
| UniformSparsity=60%, Latency=<1s, Backbone=Llama-2-13B2025.05 | 8.14 | — | — | — | |
| SliceGPTRatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 8.21 | — | — | — | |
| Flash Linear AttentionModel scale=9B2026.06 | 8.21 | — | — | — | |
| Multiplication-Only Matrix Inversion ApproximationModel scale=9B2026.06 | 8.21 | — | — | — | |
| SliceGPTRatio=20%, Backbone=Mistral, Model Scale=7B2026.03 | 8.23 | — | — | — | |
| FLAPRatio=20%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 8.76 | — | — | — | |
| GPTQModel Architecture=Llama-3-8B2025.12 | 8.81 | — | — | — | |
| BolacoRatio=30%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 8.83 | — | — | — | |
| Flash Linear AttentionModel scale=4B2026.06 | 8.89 | — | — | — | |
| Multiplication-Only Matrix Inversion ApproximationModel scale=4B2026.06 | 8.89 | — | — | — | |
| RTN#Bits (W-A-KV)=4-8-4, Model=LLaMA-30B2023.05 | 8.9 | — | — | — | |
| FLAPRatio=30%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 8.91 | — | — | — | |
| Cosine (Window)Sparsity=25%, Latency=24s, Model=Llama-2-13B2025.05 | 8.99 | — | — | — | |
| FP16Model Architecture=Llama-3.2-3B2025.12 | 9.53 | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-16-16, Model=LLaMA-13B2023.05 | 9.6 | — | — | — | |
| CALMModel Architecture=Llama-3.2-3B2025.12 | 9.63 | — | — | — | |
| LLM-PrunerRatio=20%, Backbone=LLaMA-2, Model Scale=13B2026.03 | 9.67 | — | — | — | |
| LLaMA-13B#Bits (W-A-KV)=16-16-162023.05 | 9.7 | — | — | — | |
| SliceGPTRatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 9.7 | — | — | — | |
| SpinQuantModel Architecture=Llama-3.2-3B2025.12 | 9.75 | — | — | — | |
| SmoothQuantModel Architecture=Llama-3.2-3B2025.12 | 9.8 | — | — | — | |
| GPTQModel Architecture=Llama-3.2-3B2025.12 | 9.82 | — | — | — | |
| AWQModel Architecture=Llama-3.2-3B2025.12 | 9.91 | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-8-4, Model=LLaMA-13B2023.05 | 10.2 | — | — | — | |
| EvoPressSparsity=70%, Latency=138m, Backbone=Llama-2-13B2025.05 | 10.24 | — | — | — | |
| UniCuCoSparsity=25%, Latency=<1s, Model=Llama-2-13B2025.05 | 10.39 | — | — | — | |
| LLaMA-7B#Bits (W-A-KV)=16-16-162023.05 | 10.4 | — | — | — | |
| FP16Model Architecture=Qwen1.5-1.5B2025.12 | 10.5 | — | — | — | |
| LLM-PrunerRatio=20%, Backbone=LLaMA-2, Model Scale=7B2026.03 | 10.55 | — | — | — | |
| GPTQ#Bits (W-A-KV)=4-16-16, Model=LLaMA-13B2023.05 | 10.7 | — | — | — | |
| NeUQIModel=Qwen 2.5, Size=72B, Bits=22025.05 | 10.79 | — | — | — | |
| FLAPRatio=30%, Backbone=LLaMA-2, Model Scale=70B2026.03 | 10.8 | — | — | — | |
| LLM-QAT#Bits (W-A-KV)=4-16-16, Model=LLaMA-7B2023.05 | 10.9 | — | — | — | |
| BF16 BaselineAvg. Bitwidth=16.0002025.07 | 11.01 | — | — | — | |
| Flash Linear AttentionModel scale=2B2026.06 | 11.07 | — | — | — | |
| Multiplication-Only Matrix Inversion ApproximationModel scale=2B2026.06 | 11.07 | — | — | — | |
| HPTQAvg. Bitwidth=4.1252025.07 | 11.27 | — | — | — |