Language Modeling on PTB
4.345PerplexitySGD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SGDModel Architecture=RoBERTa2026.04 | 4.345 | — | |
| AdaCubicModel Architecture=RoBERTa2026.04 | 5.145 | — | |
| AdaCubicModel Architecture=DistilBERT2026.04 | 7.334 | — | |
| AdaHessianModel Architecture=RoBERTa2026.04 | 7.582 | — | |
| BaselinePrecision=FP16, Model Size=30B2024.02 | 8.159 | — | |
| BaselinePrecision=FP16, Model Family=LLaMA, Model Size=30B2024.02 | 8.159 | — | |
| aespaPrecision=INT4, Model Size=30B2024.02 | 8.283 | — | |
| aespaPrecision=INT4, Model Family=LLaMA, Model Size=30B2024.02 | 8.283 | — | |
| SGDModel Architecture=DistilBERT2026.04 | 8.299 | — | |
| DenseCompression Ratio=1.0, Backbone=LLaMA-7B2026.04 | 8.34 | — | |
| BaselineBackbone=LLaMA-7B, Ratio=0.0, Fine-tuning=false, Mixed-rank=false2026.02 | 8.35 | — | |
| OmniQuantPrecision=INT4, Model Family=LLaMA, Model Size=30B2024.02 | 8.354 | — | |
| AffineQuantPrecision=INT4, Model Family=LLaMA, Model Size=30B2024.02 | 8.355 | — | |
| Z-FOLDPrecision=INT4, Model Size=30B2024.02 | 8.41 | — | |
| OPTQPrecision=INT4, Model Size=30B2024.02 | 8.426 | — | |
| RTNPrecision=INT4, Model Size=30B2024.02 | 8.653 | — | |
| aespaPrecision=INT3, Model Size=30B2024.02 | 8.684 | — | |
| aespaPrecision=INT3, Model Family=LLaMA, Model Size=30B2024.02 | 8.684 | — | |
| FP16 baselineBackbone Model=OPT-6.7B2023.08 | 8.76 | — | |
| LLAMA-7BPruning Ratio=0%, Fine-tuning=LaMini2023.05 | 8.93 | — | |
| AA-SVDCompression Ratio=0.8, Backbone=LLaMA-7B, Weight Remapping=Dobi-SVD-style2026.04 | 8.97 | — | |
| Z-FOLDPrecision=INT3, Model Size=30B2024.02 | 8.979 | — | |
| AffineQuantPrecision=INT3, Model Family=LLaMA, Model Size=30B2024.02 | 8.998 | — | |
| OmniQuantPrecision=INT3, Model Family=LLaMA, Model Size=30B2024.02 | 9.065 | — | |
| BaselinePrecision=FP16, Model Size=13B2024.02 | 9.081 | — | |
| BaselinePrecision=FP16, Model Family=LLaMA, Model Size=13B2024.02 | 9.081 | — | |
| aespaPrecision=INT4, Model Size=13B2024.02 | 9.277 | — | |
| aespaPrecision=INT4, Model Family=LLaMA, Model Size=13B2024.02 | 9.277 | — | |
| AffineQuantPrecision=INT4, Model Family=LLaMA, Model Size=13B2024.02 | 9.325 | — | |
| OmniQuantPrecision=INT4, Model Family=LLaMA, Model Size=13B2024.02 | 9.33 | — | |
| Z-FOLDPrecision=INT4, Model Size=13B2024.02 | 9.335 | — | |
| RFID-MoEBackbone=Qwen2-57B-A14B, Ratio=40%2026.02 | 9.46 | — | |
| OPTQPrecision=INT4, Model Size=13B2024.02 | 9.526 | — | |
| RTNPrecision=INT4, Model Size=13B2024.02 | 9.775 | — | |
| aespaPrecision=INT3, Model Size=13B2024.02 | 9.818 | — | |
| aespaPrecision=INT3, Model Family=LLaMA, Model Size=13B2024.02 | 9.818 | — | |
| DenseRatio=1.0, Base Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2026.04 | 9.89 | — | |
| FP16Backbone=LLaMA3-8B, Precision=FP162024.06 | 9.91 | — | |
| DenseBudget=100%, Base Model=Llama3-8B, Zero-shot=true2025.05 | 9.91 | — | |
| OPTQPrecision=INT3, Model Size=30B2024.02 | 9.964 | — | |
| Z-FOLDPrecision=INT3, Model Size=13B2024.02 | 10.09 | — | |
| BaselinePrecision=FP16, Model Size=7B2024.02 | 10.12 | — | |
| BaselinePrecision=FP16, Model Family=LLaMA, Model Size=7B2024.02 | 10.12 | — | |
| RFID-MoEBackbone=DeepSeekMoE-16B-Base, Ratio=40%2026.02 | 10.18 | — | |
| AdaHessianModel Architecture=DistilBERT2026.04 | 10.182 | — | |
| FP16 baselinePrecision=FP16, Model Architecture=OPT, Model Size=6.7B2023.08 | 10.21 | — | |
| OmniQuantPrecision=INT3, Model Family=LLaMA, Model Size=13B2024.02 | 10.24 | — | |
| AffineQuantPrecision=INT3, Model Family=LLaMA, Model Size=13B2024.02 | 10.24 | — | |
| aespaPrecision=INT4, Model Size=7B2024.02 | 10.43 | — | |
| aespaPrecision=INT4, Model Family=LLaMA, Model Size=7B2024.02 | 10.43 | — | |
| AffineQuantPrecision=INT4, Model Family=LLaMA, Model Size=7B2024.02 | 10.53 | — | |
| OmniQuantPrecision=INT4, Model Family=LLaMA, Model Size=7B2024.02 | 10.57 | — | |
| LLAMA-3-8BModel=LLAMA-3-8B, Bits=FP162026.04 | 10.6 | — | |
| Llama-3.1-8BCompression Ratio=Original2025.10 | 10.75 | — | |
| DuQuantBackbone=LLaMA3-8B, Weight-Activation Bits=W6A62024.06 | 10.77 | — | |
| DuQuant+LWCBackbone=LLaMA3-8B, Weight-Activation Bits=W6A62024.06 | 10.78 | — | |
| TSLDPrecision=W4A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11 | — | |
| AA-SVDCompression Ratio=0.6, Backbone=LLaMA-7B, Weight Remapping=Dobi-SVD-style2026.04 | 11.07 | — | |
| TSLDPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.17 | — | |
| LogitPrecision=W4A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.2 | — | |
| RTNPrecision=INT4, Model Size=7B2024.02 | 11.25 | — | |
| LLaMA3Size=3.0B, Bit=16.02025.08 | 11.3 | — | |
| LogitPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.33 | — | |
| DenseBudget=100%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 11.36 | — | |
| DenseRatio=1.0, zero-shot evaluation=true2026.04 | 11.37 | — | |
| FP16 baselinePrecision=FP16, Model Architecture=OPT, Model Size=2.7B2023.08 | 11.43 | — | |
| NeuronMLPBase Model=Llama-3.1-8B, Compression Ratio=0.052025.10 | 11.43 | — | |
| Z-FOLDPrecision=INT4, Model Size=7B2024.02 | 11.45 | — | |
| aespaPrecision=INT3, Model Size=7B2024.02 | 11.45 | — | |
| aespaPrecision=INT3, Model Family=LLaMA, Model Size=7B2024.02 | 11.45 | — | |
| OPTQPrecision=INT4, Model Size=7B2024.02 | 11.51 | — | |
| TSLDPrecision=W4A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=2.7B2023.08 | 11.59 | — | |
| TSLDBackbone Model=OPT-6.7B2023.08 | 11.6 | — | |
| SmoothQuantBackbone=LLaMA3-8B, Weight-Activation Bits=W6A62024.06 | 11.69 | — | |
| OPTQPrecision=W4A16, Quantization Method=PTQ, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.73 | — | |
| Z-FOLDPrecision=INT3, Model Size=7B2024.02 | 11.73 | — | |
| L2L+LogitPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.75 | — | |
| Logit+GTPrecision=W4A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 11.78 | — | |
| Llama-3.2-3BCompression Ratio=Original2025.10 | 11.78 | — | |
| OriginalBase Model=Deepseek-V2-Lite-Chat, Compression Ratio=None2026.02 | 11.79 | — | |
| OPTQPrecision=W4A16, Quantization Method=PTQ, Model Architecture=OPT, Model Size=2.7B2023.08 | 11.82 | — | |
| LogitPrecision=W4A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=2.7B2023.08 | 11.82 | — | |
| LoRAPrunePruning Ratio=20%, Fine-tuning=LaMini2023.05 | 11.87 | — | |
| OmniQuantBackbone=LLaMA3-8B, Weight-Activation Bits=W6A62024.06 | 11.9 | — | |
| RFID-MoEBase Model=Deepseek-V2-Lite-Chat, Compression Ratio=40%2026.02 | 11.9 | — | |
| LoRAPrune-8bitPruning Ratio=20%, Fine-tuning=LaMini, Precision=8-bit2023.05 | 11.91 | — | |
| OPTQPrecision=INT3, Model Size=13B2024.02 | 11.91 | — | |
| AffineQuantPrecision=INT3, Model Family=LLaMA, Model Size=7B2024.02 | 11.92 | — | |
| DenseBudget=100%¹, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 11.93 | — | |
| TSLDPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=2.7B2023.08 | 11.97 | — | |
| OmniQuantPrecision=INT3, Model Family=LLaMA, Model Size=7B2024.02 | 11.98 | — | |
| RFID-MoEBackbone=Qwen1.5-MoE-A2.7B, Ratio=40%2026.02 | 11.98 | — | |
| MoBEBase Model=Deepseek-V2-Lite-Chat, Compression Ratio=40%2026.02 | 12.17 | — | |
| Logit KDBackbone Model=OPT-6.7B2023.08 | 12.22 | — | |
| Qwen-3-8BCompression Ratio=Original2025.10 | 12.25 | — | |
| LogitPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=2.2B2023.08 | 12.26 | — | |
| AffineQuantBackbone=LLaMA3-8B, Weight-Activation Bits=W6A62024.06 | 12.3 | — | |
| AA-SVDCompression Ratio=0.8, Backbone=LLaMA-7B2026.04 | 12.3 | — | |
| LLM-PrunerPruning Ratio=20%, Fine-tuning=LaMini2023.05 | 12.38 | — | |
| Logit+GTPrecision=W2A16, Quantization Method=QAT, Model Architecture=OPT, Model Size=6.7B2023.08 | 12.41 | — |