Language Modeling on WikiText-2 (val) (Sparsity Evaluation)
20PPL (10% sparsity)co-abl. static
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| co-abl. staticModel=Qwen-2.5-7B, Dense PPL=16.22026.07 | 20 | 27 | 31 | 50 | 91 | 426 | 1,341 | |
| COAXModel=Qwen-2.5-7B, Dense PPL=16.22026.07 | 21 | 24 | 29 | 35 | 54 | 92 | 240 | |
| WandaModel=Pythia-1.4B, Dense PPL=21.92026.07 | 22 | 25 | 149 | 171 | 356 | 450 | 558 | |
| magnitudeModel=Pythia-1.4B, Dense PPL=21.92026.07 | 23 | 24 | 28 | 78 | 178 | 217 | 459 | |
| TaylorModel=Pythia-1.4B, Dense PPL=21.92026.07 | 23 | 24 | 26 | 31 | 41 | 55 | 99 | |
| co-abl. staticModel=Pythia-1.4B, Dense PPL=21.92026.07 | 23 | 24 | 25 | 30 | 38 | 46 | 66 | |
| COAXModel=Pythia-1.4B, Dense PPL=21.92026.07 | 23 | 23 | 25 | 29 | 35 | 47 | 73 | |
| randomModel=Pythia-1.4B, Dense PPL=21.92026.07 | 25 | 38 | 108 | 158 | 491 | 1,155 | 1,302 | |
| magnitudeModel=GPT-2-large, Dense PPL=29.62026.07 | 30 | 32 | 38 | 51 | 77 | 124 | 184 | |
| TaylorModel=GPT-2-large, Dense PPL=29.62026.07 | 30 | 33 | 36 | 41 | 45 | 55 | 78 | |
| co-abl. staticModel=GPT-2-large, Dense PPL=29.62026.07 | 30 | 31 | 32 | 36 | 44 | 55 | 91 | |
| COAXModel=GPT-2-large, Dense PPL=29.62026.07 | 30 | 31 | 32 | 34 | 37 | 42 | 52 | |
| randomModel=GPT-2-large, Dense PPL=29.62026.07 | 31 | 34 | 40 | 45 | 117 | 277 | 488 | |
| WandaModel=GPT-2-large, Dense PPL=29.62026.07 | 31 | 37 | 63 | 88 | 210 | 316 | 653 | |
| randomModel=Qwen-2.5-7B, Dense PPL=16.22026.07 | 46 | 173 | 601 | 1,195 | 7,544 | 18,298 | 119,377 | |
| WandaModel=Qwen-2.5-7B, Dense PPL=16.22026.07 | 46 | 66 | 29,709 | 66,182 | 79,750 | 80,023 | 78,816 | |
| magnitudeModel=Qwen-2.5-7B, Dense PPL=16.22026.07 | 49 | 66 | 123 | 29,613 | 65,157 | 80,280 | 79,277 |