Language Modeling on WikiText-2 (Perplexity, Improvement)
4.74PerplexityDense
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DenseModel=Mistral-7B, Sparsity=0%2026.07 | 4.74 | — | |
| DenseModel=LLaMA-2-7B, Sparsity=0%2026.07 | 5.33 | — | |
| DenseModel=LLaMA-3-8B, Sparsity=0%2026.07 | 5.46 | — | |
| WandaModel=Mistral-7B, Sparsity=50%2026.07 | 6.31 | — | |
| PALSModel=Mistral-7B, Sparsity=50%2026.07 | 6.31 | 0 | |
| PALSModel=LLaMA-3-8B, Sparsity=50%2026.07 | 10.45 | 0.3 | |
| WandaModel=LLaMA-3-8B, Sparsity=50%2026.07 | 10.48 | — | |
| PALSModel=LLaMA-2-7B, Sparsity=50%2026.07 | 10.96 | 15.2 | |
| WandaModel=LLaMA-2-7B, Sparsity=50%2026.07 | 12.92 | — |