List your company
ResearchFormerly WizWand

Language Modeling on Penn Treebank (PTB)

7.11PPLLoRA
4.65421.23237.8154.388May 23, 2023Nov 19, 2023May 17, 2024Nov 14, 2024May 13, 2025Nov 9, 2025May 9, 2026
Updated 4mo ago

Evaluation results

MethodLinks
LoRAW Bits=16, Model=LLaMA 65B2023.057.11
LoRAW Bits=16, Model=LLaMA 30B2023.057.21
PEQAW Bits=4, Model=LLaMA 65B2023.057.36
LoRA+OPTQW Bits=4, Model=LLaMA 65B2023.057.46
LoRA+OPTQW Bits=4, Model=LLaMA 30B2023.057.55
PEQAW Bits=4, Model=LLaMA 30B2023.057.68
BaselineCompression Ratio=0.02026.058.35
LoRAW Bits=16, Model=LLaMA 13B2023.058.52
PEQAW Bits=4, Model=LLaMA 13B2023.058.64
LoRA+OPTQW Bits=4, Model=LLaMA 13B2023.058.83
LoRAW Bits=16, Model=LLaMA 7B2023.059.14
OriginalRatio=0%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.099.18
HEAPrRatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.099.34
OriginalRatio=0%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.099.47
PEQAW Bits=4, Model=LLaMA 7B2023.059.69
HEAPrRatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.099.88
MoE-SVDRatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0910.48
HEAPrRatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0910.86
D^2-MoERatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0911.1
Meta-Llama-3-8B Dense2026.0311.18
LoRA+OPTQW Bits=4, Model=LLaMA 7B2023.0511.22
D^2-MoERatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.0911.23
NAEERatio=40%, Backbone=Qwen2-57B-A14B, Evaluation Protocol=zero-shot2025.0911.34
MoE-I^2Ratio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0911.59
LoRAW Bits=16, Model=GPT-J 6B2023.0512.92
OriginalRatio=0%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0912.97
PEQAW Bits=4, Model=GPT-J 6B2023.0513.3
LoRA+OPTQW Bits=4, Model=GPT-J 6B2023.0513.46
3BASiL-TMConfig=4:8+64LR, One-shot=true2026.0313.7
Dobi-SVD + PARSECompression Ratio=0.22026.0513.9
SAES-SVD + PARSECompression Ratio=0.22026.0513.96
DenseModel=OPT-30B2026.0314.04
D^2-MoERatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0914.07
HEAPrRatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0914.12
3BASiLConfig=4:8+64LR, One-shot=true2026.0314.24
3BASiLModel=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=4.202026.0314.26
Hf-ALPS-10Model=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=4.332026.0314.33
SVD-LLM + PARSECompression Ratio=0.22026.0514.37
Hf-SparseGPTModel=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=5.972026.0314.39
Hassle-free-ALPSConfig=4:8+64LR, One-shot=true2026.0314.45
NAEERatio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0914.47
OATS-10Model=OPT-30B, Decomposition Configuration=2:4 + 112, Compression Time (hrs)=5.812026.0314.65
Dobi-SVDCompression Ratio=0.22026.0514.83
Sub-MoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0914.84
3BASiL-TMConfig=2:4+64LR, One-shot=true2026.0314.88
Basis Sharing + PARSECompression Ratio=0.22026.0514.98
NAEERatio=20%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0915.02
SAES-SVDCompression Ratio=0.22026.0515.16
Hassle-free-SparseGPTConfig=4:8+64LR, One-shot=true2026.0315.21
OriginalRatio=0%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0915.4
MoE-I^2Ratio=40%, Backbone=DeepSeekMoE-16B-Base, Evaluation Protocol=zero-shot2025.0915.75
LoRAW Bits=16, Model=GPT-Neo 2.7B2023.0515.92
SVD-LLMCompression Ratio=0.22026.0516.22
3BASiLConfig=2:4+64LR, One-shot=true2026.0316.25
HC-SMoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0916.39
PEQAW Bits=4, Model=GPT-Neo 2.7B2023.0516.55
ASVDCompression Ratio=0.22026.0516.55
Hassle-free-ALPSConfig=2:4+64LR, One-shot=true2026.0316.71
HEAPrRatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0916.8
OATSConfig=4:8+64LR, One-shot=true2026.0317.23
Basis SharingCompression Ratio=0.22026.0517.35
MC-SMoERatio=25%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0917.45
HEAPrRatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0917.58
Hassle-free-SparseGPTConfig=2:4+64LR, One-shot=true2026.0318.71
LoRA+OPTQW Bits=4, Model=GPT-Neo 2.7B2023.0518.83
3BASiL-TMConfig=3:8+64LR, One-shot=true2026.0320.47
LLaMA-7B*Ratio=0%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0322.14
OATSConfig=2:4+64LR, One-shot=true2026.0323.41
Sub-MoERatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0923.48
3BASiLConfig=3:8+64LR, One-shot=true2026.0324.84
HEAPrRatio=50%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0926.29
Hassle-free-ALPSConfig=3:8+64LR, One-shot=true2026.0326.31
OriginalParam Ratio=1.0, Backbone=LLaMA-2-7B2026.0427.028
LoRAPruneRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0328.75
Sub-MoERatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0929
HC-SMoERatio=25%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0931.11
HyWIARatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0331.16
SLEBRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0331.6
Hassle-free-SparseGPTConfig=3:8+64LR, One-shot=true2026.0332.06
WANDA*Ratio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0333.16
FLAPParam Ratio=0.8, Backbone=LLaMA-2-7B2026.0436.9429
HC-SMoERatio=50%, Backbone=Qwen1.5-MoE-A2.7B-Chat, Evaluation Protocol=zero-shot2025.0938.18
MagnitudeRatio=20%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0338.64
Dobi-SVD + PARSECompression Ratio=0.42026.0540.16
SAES-SVD + PARSECompression Ratio=0.42026.0542.13
Sub-MoERatio=50%, Backbone=Qwen3-30B-A3B, Evaluation Protocol=zero-shot2025.0943.19
HyWIARatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0344.38
Basis Sharing + PARSECompression Ratio=0.42026.0544.49
SAES-SVDCompression Ratio=0.42026.0545.13
Dobi-SVDCompression Ratio=0.42026.0546.38
GRASPruneParam Ratio=0.8, Backbone=LLaMA-2-7B2026.0448.1827
GRASPruneModel=LLaMA-7B, Parameter Retention=40%2026.0449.7172
SVD-LLM + PARSECompression Ratio=0.42026.0550.17
LoRAPruneRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0350.3
LLM-PrunerParam Ratio=0.8, Backbone=LLaMA-2-7B2026.0453.1254
LoRAPrune-8bitRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0353.24
Basis SharingCompression Ratio=0.42026.0555.78
SVD-LLMCompression Ratio=0.42026.0563.75
OATSConfig=3:8+64LR, One-shot=true2026.0367.35
VectorRatio=50%, Zero-shot evaluation=true, Model Backbone=LLaMA-7B2024.0368.51