List your company
ResearchFormerly WizWand

Language Modeling on Lambada, C4, WikiText2 Aggregate

1PPL DegradationDense
-12.532478.8113170.155261.4987Jun 1, 2026
Updated 5mo ago

Evaluation results

MethodLinks
DenseSparsity=0%, Backbone=Llama-3.2-3B2026.061
SUBFITSparsity=12.5%2026.061.4
ReplaceMe (Cosine)Sparsity=12.5%2026.061.51
Streamline (Layer)Sparsity=12.5%2026.061.52
ReplaceMe (LS)Sparsity=12.5%2026.061.58
Streamline (FFN)Sparsity=12.5%2026.061.62
SUBFITBase Model=Llama-3.1-8B, Sparsity Level=20%2026.061.66
SUBFITBackbone=Llama-3.1-8B, Sparsity=20%2026.061.66
SUBFITSparsity=20%, Backbone=Llama-3.1-8B2026.061.66
SUBFITBase Model=Llama-3.2-3B, Sparsity Level=20%2026.061.69
SUBFITSparsity=20%, Backbone=Llama-3.2-3B2026.061.69
SUBFITBase Model=DeepSeek-7B, Sparsity Level=20%2026.061.72
SUBFITBackbone=DeepSeek-7B, Sparsity=20%2026.061.72
SUBFITSparsity=20%, Backbone=Qwen3-4B2026.061.8
SUBFITSparsity=20.0%2026.061.81
SUBFITBase Model=Qwen3-8B, Sparsity Level=20%2026.062.25
SUBFITSparsity=25.0%2026.062.42
Streamline (FFN)Sparsity=20.0%2026.062.43
ReplaceMe (Cosine)Sparsity=20.0%2026.062.46
Streamline (Layer)Sparsity=20.0%2026.062.5
SUBFITBase Model=Llama-3.2-3B, Sparsity Level=30%2026.062.63
SUBFITSparsity=30%, Backbone=Llama-3.2-3B2026.062.63
ReplaceMe (LS)Sparsity=20.0%2026.062.82
SUBFITBase Model=Llama-3.1-8B, Sparsity Level=30%2026.063
SUBFITSparsity=30%, Backbone=Llama-3.1-8B2026.063
SUBFITBase Model=DeepSeek-7B, Sparsity Level=30%2026.063.04
SUBFITSparsity=30.0%2026.063.22
SUBFITBase Model=Qwen3-4B, Sparsity Level=30%2026.063.5
SUBFITSparsity=30%, Backbone=Qwen3-4B2026.063.5
SUBFITBase Model=Qwen3-8B, Sparsity Level=30%2026.064.11
SUBFITBackbone=Qwen3-8B, Sparsity=30%2026.064.11
ReplaceMe (Cosine)Sparsity=25.0%2026.064.34
ReplaceMe (LS)Sparsity=25.0%2026.064.67
SUBFITBase Model=DeepSeek-7B, Sparsity Level=37.5%2026.064.74
SUBFITBase Model=Llama-3.2-3B, Sparsity Level=37.5%2026.064.85
SUBFITBackbone=Llama-3.2-3B, Sparsity=37.5%2026.064.85
SUBFITSparsity=37.5%, Backbone=Llama-3.2-3B2026.064.85
SUBFITBase Model=Llama-3.1-8B, Sparsity Level=37.5%2026.065.65
SUBFITSparsity=37.5%, Backbone=Llama-3.1-8B2026.065.65
SUBFITSparsity=37.5%, Backbone=Qwen3-4B2026.065.75
SUBFITSparsity=37.5%2026.065.76
ReplaceMe (Cosine)Base Model=Qwen3-8B, Sparsity Level=37.5%2026.066.59
Streamline (Layer)Sparsity=25.0%2026.066.61
Streamline (Layer)Sparsity=30.0%2026.066.61
Streamline (FFN)Sparsity=30.0%2026.067.06
ReplaceMe (Cosine)Sparsity=30.0%2026.067.38
ReplaceMe (LS)Sparsity=30.0%2026.068.09
Streamline (FFN)Sparsity=25.0%2026.068.31
ReplaceMe (LS)Sparsity=37.5%2026.0611.45
Streamline (Layer)Sparsity=37.5%2026.0612.44
Streamline (FFN)Sparsity=37.5%2026.0616.01
ReplaceMe (Cosine)Sparsity=37.5%2026.0622.28
ReplaceMe (Cosine)Backbone=Qwen3-4B, Sparsity=37.5%2026.06339.31