Language Modeling on WikiText 1,000-example evaluation slice (test)
12.723PerplexityAdditive sparse
Evaluation results
| Method | Links | |
|---|---|---|
| Additive sparseBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=KL2026.05 | 12.723 | |
| Additive sparseBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=TF-IDF2026.05 | 12.773 | |
| Additive sparse +SBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=KL2026.05 | 13.139 | |
| Base QwenBackbone=Qwen-2.5-0.5B-Instruct2026.05 | 13.146 | |
| Additive sparse +SBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=TF-IDF2026.05 | 13.456 | |
| LoRABackbone=Qwen-2.5-0.5B-Instruct2026.05 | 15.47 | |
| Replacement sparseBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=TF-IDF2026.05 | 16.501 | |
| Replacement sparseBackbone=Qwen-2.5-0.5B-Instruct, Selection rule=KL2026.05 | 16.577 | |
| Full finetuneBackbone=Qwen-2.5-0.5B-Instruct2026.05 | 18.907 |