Language Modeling on Penn Treebank
11.18PerplexityMeta-Llama-3-8B Dense
Evaluation Results
| Method | Links | |
|---|---|---|
| Meta-Llama-3-8B DenseConfig=Dense, Fine-tuning Protocol=None, Base Model=Meta-Llama-3-8B2026.03 | 11.18 | |
| LFT-3BASiL-TMConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 13.41 | |
| LFT-Hassle-free-ALPS-TMConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 13.43 | |
| LFT-3BASiLConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 13.43 | |
| LFT-Hassle-free-SparseGPT-TMConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 13.71 | |
| LFT-Hassle-free-SparseGPTConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.03 | |
| LFT-3BASiL-TMConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.17 | |
| LFT-Hassle-free-ALPS-TMConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.34 | |
| LFT-Hassle-free-ALPSConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.45 | |
| LFT-3BASiLConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.54 | |
| LFT-Hassle-free-ALPSConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.61 | |
| LFT-OATSConfig=4:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.64 | |
| LFT-OATS-TMConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.81 | |
| LFT-Hassle-free-SparseGPT-TMConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 14.84 | |
| LFT-Hassle-free-SparseGPTConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 15.35 | |
| LFT-OATSConfig=2:4+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 16.24 | |
| LFT-3BASiL-TMConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 17.27 | |
| Llama-3.2-1B DenseConfig=Dense2026.03 | 17.59 | |
| LFT-Hassle-free-ALPS-TMConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 17.74 | |
| LFT-3BASiLConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 17.85 | |
| LFT-Hassle-free-ALPSConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 17.89 | |
| LFT-Hassle-free-SparseGPT-TMConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 18.08 | |
| LFT-OATS-TMConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 18.34 | |
| LFT-Hassle-free-SparseGPTConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 18.84 | |
| LLaMA-13B (Ma, Fang, and Wang 2023)Pruning Ratio=0%, Zero-shot=true2024.03 | 20.24 | |
| LFT-OATSConfig=3:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 20.59 | |
| LFT-3BASiL-TMConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 22.3 | |
| LFT-3BASiLConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 22.56 | |
| LFT-Hassle-free-ALPS-TMConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 22.75 | |
| LFT-Hassle-free-SparseGPT-TMConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 23.6 | |
| LFT-Hassle-free-ALPSConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 23.63 | |
| LFT-OATS-TMConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 23.65 | |
| LFT-3BASiL-TMConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 23.88 | |
| LFT-Hassle-free-ALPS-TMConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 24.31 | |
| LFT-3BASiLConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 24.56 | |
| LFT-Hassle-free-SparseGPTConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 24.74 | |
| LFT-Hassle-free-SparseGPT-TMConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 25.23 | |
| LFT-Hassle-free-ALPSConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 25.47 | |
| LFT-OATS-TMConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 25.77 | |
| LFT-Hassle-free-SparseGPTConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 26.81 | |
| HyWIAPruning Ratio=20%, Zero-shot=true2024.03 | 27.55 | |
| LFT-Hassle-free-ALPS-TMConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 28 | |
| Block (Ma, Fang, and Wang 2023)Pruning Ratio=20%, Zero-shot=true2024.03 | 28.08 | |
| LFT-3BASiL-TMConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 29 | |
| LFT-OATSConfig=2:4+64LR, Fine-Tuning=LoRA2026.03 | 29.6 | |
| LFT-3BASiLConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 29.74 | |
| LFT-3BASiL-TMConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 29.94 | |
| LFT-3BASiLConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 30.05 | |
| LFT-Hassle-free-ALPS-TMConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 30.32 | |
| LFT-Hassle-free-SparseGPT-TMConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 31.65 | |
| LFT-Hassle-free-ALPSConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 31.99 | |
| LFT-Hassle-free-ALPSConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 32.26 | |
| LFT-Hassle-free-SparseGPT-TMConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 32.36 | |
| LFT-OATSConfig=4:8+64LR, Fine-Tuning=LoRA2026.03 | 32.9 | |
| LFT-Hassle-free-SparseGPTConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 33.94 | |
| LFT-Hassle-free-SparseGPTConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 34.55 | |
| LFT-OATS-TMConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 34.83 | |
| LFT-OATS-TMConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 36.25 | |
| L2 (Ma, Fang, and Wang 2023)Pruning Ratio=20%, Zero-shot=true2024.03 | 38.05 | |
| LFT-OATSConfig=3:8+64LR, Fine-Tuning=LoRA2026.03 | 41.74 | |
| LFT-3BASiLConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 45.13 | |
| LFT-3BASiL-TMConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 47.11 | |
| LFT-OATSConfig=2:8+64LR, Fine-tuning Protocol=LFT, Base Model=Meta-Llama-3-8B2026.03 | 49.71 | |
| LFT-Hassle-free-ALPS-TMConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 50.15 | |
| LFT-Hassle-free-SparseGPT-TMConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 55.53 | |
| LFT-Hassle-free-ALPSConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 57.45 | |
| LFT-Hassle-free-SparseGPTConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 63.87 | |
| LFT-OATS-TMConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 68.98 | |
| LFT-OATSConfig=2:8+64LR, Fine-Tuning=LoRA2026.03 | 95.05 |