Language Modeling on Instruction-tuned models aggregate
93.4Accuracy RetentionSUBFIT
Evaluation Results
| Method | Links | |
|---|---|---|
| SUBFITSparsity=12.5%2026.06 | 93.4 | |
| Streamline (Layer)Sparsity=12.5%2026.06 | 93.2 | |
| ReplaceMe (Cosine)Sparsity=12.5%2026.06 | 93.2 | |
| ReplaceMe (LS)Sparsity=12.5%2026.06 | 92.7 | |
| Streamline (FFN)Sparsity=12.5%2026.06 | 91.7 | |
| SUBFITSparsity=20.0%2026.06 | 88.1 | |
| ReplaceMe (Cosine)Sparsity=20.0%2026.06 | 86.8 | |
| Streamline (FFN)Sparsity=20.0%2026.06 | 86.2 | |
| ReplaceMe (LS)Sparsity=20.0%2026.06 | 85.8 | |
| Streamline (Layer)Sparsity=20.0%2026.06 | 85.4 | |
| SUBFITSparsity=25.0%2026.06 | 84.6 | |
| Streamline (Layer)Sparsity=25.0%2026.06 | 81.6 | |
| ReplaceMe (LS)Sparsity=25.0%2026.06 | 81.5 | |
| ReplaceMe (Cosine)Sparsity=25.0%2026.06 | 81.5 | |
| SUBFITSparsity=30.0%2026.06 | 80.8 | |
| ReplaceMe (Cosine)Sparsity=30.0%2026.06 | 79.1 | |
| ReplaceMe (LS)Sparsity=30.0%2026.06 | 77.7 | |
| Streamline (FFN)Sparsity=25.0%2026.06 | 77.2 | |
| Streamline (Layer)Sparsity=30.0%2026.06 | 74.9 | |
| SUBFITSparsity=37.5%2026.06 | 73.9 | |
| Streamline (FFN)Sparsity=30.0%2026.06 | 72.3 | |
| ReplaceMe (Cosine)Sparsity=37.5%2026.06 | 70.9 | |
| ReplaceMe (LS)Sparsity=37.5%2026.06 | 70.8 | |
| Streamline (Layer)Sparsity=37.5%2026.06 | 68.7 | |
| Streamline (FFN)Sparsity=37.5%2026.06 | 64.8 |