Language Modeling on WikiText-103 (fine-tuned)
5.95Perplexity (PPL)LTE
Evaluation results
| Method | Links | |
|---|---|---|
| LTEOverall Sparsity=52%, Calibration Data=Wiki, Backbone Model=LLaMA-7B, η=22024.02 | 5.95 | |
| Wanda (Unstructured)Overall Sparsity=50%, Calibration Data=Wiki, Backbone Model=LLaMA-7B, Pruning Type=Unstructured2024.02 | 6.87 | |
| Wanda (Unstructured)Overall Sparsity=50%, Calibration Data=C4, Backbone Model=LLaMA-7B, Pruning Type=Unstructured2024.02 | 7.04 | |
| Wanda (4:8)Overall Sparsity=50%, Calibration Data=Wiki, Backbone Model=LLaMA-7B, Pruning Type=4:82024.02 | 8.17 | |
| Wanda (4:8)Overall Sparsity=50%, Calibration Data=C4, Backbone Model=LLaMA-7B, Pruning Type=4:82024.02 | 8.39 | |
| Wanda (2:4)Overall Sparsity=50%, Calibration Data=Wiki, Backbone Model=LLaMA-7B, Pruning Type=2:42024.02 | 10.82 | |
| Wanda (2:4)Overall Sparsity=50%, Calibration Data=C4, Backbone Model=LLaMA-7B, Pruning Type=2:42024.02 | 11.45 |