Language Modeling on Penn Treebank (PTB)
7.11PPLLoRA
Evaluation Results
| Method | Links | |
|---|---|---|
| LoRAW Bits=16, Model=LLaMA 65B2023.05 | 7.11 | |
| LoRAW Bits=16, Model=LLaMA 30B2023.05 | 7.21 | |
| PEQAW Bits=4, Model=LLaMA 65B2023.05 | 7.36 | |
| LoRA+OPTQW Bits=4, Model=LLaMA 65B2023.05 | 7.46 | |
| LoRA+OPTQW Bits=4, Model=LLaMA 30B2023.05 | 7.55 | |
| PEQAW Bits=4, Model=LLaMA 30B2023.05 | 7.68 | |
| BaselineCompression Ratio=0.02026.05 | 8.35 | |
| LoRAW Bits=16, Model=LLaMA 13B2023.05 | 8.52 | |
| PEQAW Bits=4, Model=LLaMA 13B2023.05 | 8.64 | |
| LoRA+OPTQW Bits=4, Model=LLaMA 13B2023.05 | 8.83 | |
| LoRAW Bits=16, Model=LLaMA 7B2023.05 | 9.14 | |
| OriginalRatio=0%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.09 | 9.18 | |
| HEAPrRatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.09 | 9.34 | |
| OriginalRatio=0%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 9.47 | |
| PEQAW Bits=4, Model=LLaMA 7B2023.05 | 9.69 | |
| HEAPrRatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 9.88 | |
| MoE-SVDRatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 10.48 | |
| HEAPrRatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 10.86 | |
| D^2-MoERatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 11.1 | |
| Meta-Llama-3-8B Dense2026.03 | 11.18 | |
| LoRA+OPTQW Bits=4, Model=LLaMA 7B2023.05 | 11.22 | |
| D^2-MoERatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.09 | 11.23 | |
| NAEERatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.09 | 11.34 | |
| MoE-I^2Ratio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 11.59 | |
| LoRAW Bits=16, Model=GPT-J 6B2023.05 | 12.92 | |
| OriginalRatio=0%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 12.97 | |
| PEQAW Bits=4, Model=GPT-J 6B2023.05 | 13.3 | |
| LoRA+OPTQW Bits=4, Model=GPT-J 6B2023.05 | 13.46 | |
| 3BASiL-TMConfig=4:8+64LR, One-shot=true2026.03 | 13.7 | |
| Dobi-SVD + PARSECompression Ratio=0.22026.05 | 13.9 | |
| SAES-SVD + PARSECompression Ratio=0.22026.05 | 13.96 | |
| DenseModel=OPT-30B2026.03 | 14.04 | |
| D^2-MoERatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 14.07 | |
| HEAPrRatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 14.12 | |
| 3BASiLConfig=4:8+64LR, One-shot=true2026.03 | 14.24 | |
| 3BASiLModel=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=4.202026.03 | 14.26 | |
| Hf-ALPS-10Model=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=4.332026.03 | 14.33 | |
| SVD-LLM + PARSECompression Ratio=0.22026.05 | 14.37 | |
| Hf-SparseGPTModel=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=5.972026.03 | 14.39 | |
| Hassle-free-ALPSConfig=4:8+64LR, One-shot=true2026.03 | 14.45 | |
| NAEERatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 14.47 | |
| OATS-10Model=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=5.812026.03 | 14.65 | |
| Dobi-SVDCompression Ratio=0.22026.05 | 14.83 | |
| Sub-MoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 14.84 | |
| 3BASiL-TMConfig=2:4+64LR, One-shot=true2026.03 | 14.88 | |
| Basis Sharing + PARSECompression Ratio=0.22026.05 | 14.98 | |
| NAEERatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 15.02 | |
| SAES-SVDCompression Ratio=0.22026.05 | 15.16 | |
| Hassle-free-SparseGPTConfig=4:8+64LR, One-shot=true2026.03 | 15.21 | |
| OriginalRatio=0%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 15.4 | |
| MoE-I^2Ratio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.09 | 15.75 | |
| LoRAW Bits=16, Model=GPT-Neo 2.7B2023.05 | 15.92 | |
| SVD-LLMCompression Ratio=0.22026.05 | 16.22 | |
| 3BASiLConfig=2:4+64LR, One-shot=true2026.03 | 16.25 | |
| HC-SMoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 16.39 | |
| PEQAW Bits=4, Model=GPT-Neo 2.7B2023.05 | 16.55 | |
| ASVDCompression Ratio=0.22026.05 | 16.55 | |
| Hassle-free-ALPSConfig=2:4+64LR, One-shot=true2026.03 | 16.71 | |
| HEAPrRatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 16.8 | |
| OATSConfig=4:8+64LR, One-shot=true2026.03 | 17.23 | |
| Basis SharingCompression Ratio=0.22026.05 | 17.35 | |
| MC-SMoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 17.45 | |
| HEAPrRatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 17.58 | |
| Hassle-free-SparseGPTConfig=2:4+64LR, One-shot=true2026.03 | 18.71 | |
| LoRA+OPTQW Bits=4, Model=GPT-Neo 2.7B2023.05 | 18.83 | |
| 3BASiL-TMConfig=3:8+64LR, One-shot=true2026.03 | 20.47 | |
| LLaMA-7B*Ratio=0%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 22.14 | |
| OATSConfig=2:4+64LR, One-shot=true2026.03 | 23.41 | |
| Sub-MoERatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 23.48 | |
| 3BASiLConfig=3:8+64LR, One-shot=true2026.03 | 24.84 | |
| HEAPrRatio=50%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 26.29 | |
| Hassle-free-ALPSConfig=3:8+64LR, One-shot=true2026.03 | 26.31 | |
| OriginalParam Ratio=1.0, Backbone=LLaMA-2-7B2026.04 | 27.028 | |
| LoRAPruneRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 28.75 | |
| Sub-MoERatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 29 | |
| HC-SMoERatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 31.11 | |
| HyWIARatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 31.16 | |
| SLEBRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 31.6 | |
| Hassle-free-SparseGPTConfig=3:8+64LR, One-shot=true2026.03 | 32.06 | |
| WANDA*Ratio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 33.16 | |
| FLAPParam Ratio=0.8, Backbone=LLaMA-2-7B2026.04 | 36.9429 | |
| HC-SMoERatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.09 | 38.18 | |
| MagnitudeRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 38.64 | |
| Dobi-SVD + PARSECompression Ratio=0.42026.05 | 40.16 | |
| SAES-SVD + PARSECompression Ratio=0.42026.05 | 42.13 | |
| Sub-MoERatio=50%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.09 | 43.19 | |
| HyWIARatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 44.38 | |
| Basis Sharing + PARSECompression Ratio=0.42026.05 | 44.49 | |
| SAES-SVDCompression Ratio=0.42026.05 | 45.13 | |
| Dobi-SVDCompression Ratio=0.42026.05 | 46.38 | |
| GRASPruneParam Ratio=0.8, Backbone=LLaMA-2-7B2026.04 | 48.1827 | |
| GRASPruneModel=LLaMA-7B, Parameter Retention=40%2026.04 | 49.7172 | |
| SVD-LLM + PARSECompression Ratio=0.42026.05 | 50.17 | |
| LoRAPruneRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 50.3 | |
| LLM-PrunerParam Ratio=0.8, Backbone=LLaMA-2-7B2026.04 | 53.1254 | |
| LoRAPrune-8bitRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 53.24 | |
| Basis SharingCompression Ratio=0.42026.05 | 55.78 | |
| SVD-LLMCompression Ratio=0.42026.05 | 63.75 | |
| OATSConfig=3:8+64LR, One-shot=true2026.03 | 67.35 | |
| VectorRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.03 | 68.51 |