List your company
ResearchFormerly WizWand

Language Modeling on C4 (Perplexity and Perplexity Degradation)

9.36PerplexityDense
-4.188487.2633178.715270.1667May 19, 2025Jul 21, 2025Sep 22, 2025Nov 24, 2025Jan 26, 2026Mar 30, 2026Jun 1, 2026
Updated 5mo ago

Evaluation results

MethodLinks
DenseModel=Llama-3.1-8B, Sparsity=0%2026.069.361
A3Compression=10%, Model=phi-3-medium-4k-instruct (14B)2025.059.480.67
DenseModel=DeepSeek-7B, Sparsity=0%2026.069.751
SVD-LLMCompression=10%, Model=phi-3-medium-4k-instruct (14B)2025.0510.471.66
A3Compression=20%, Model=phi-3-medium-4k-instruct (14B)2025.0510.591.79
DenseModel=Llama-3.2-3B, Sparsity=0%2026.0611.071
SVD-LLMCompression=20%, Model=phi-3-medium-4k-instruct (14B)2025.0511.93.1
SUBFITModel=Llama-3.1-8B, Sparsity=12.5%2026.0613.571.4
Streamline (Layer)Model=Llama-3.1-8B, Sparsity=12.5%2026.0613.751.41
Streamline (FFN)Model=Llama-3.1-8B, Sparsity=12.5%2026.0614.071.44
ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=12.5%2026.0614.121.42
ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=12.5%2026.0614.161.42
ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=12.5%2026.0614.191.43
ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=12.5%2026.0614.261.46
Streamline (Layer)Model=DeepSeek-7B, Sparsity=12.5%2026.0614.471.46
SUBFITModel=DeepSeek-7B, Sparsity=12.5%2026.0614.471.52
Streamline (FFN)Model=DeepSeek-7B, Sparsity=12.5%2026.0615.161.58
DenseModel=Qwen3-8B, Sparsity=0%2026.0615.251
SUBFITModel=Llama-3.2-3B, Sparsity=12.5%2026.0615.961.39
ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=12.5%2026.0618.341.57
ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=12.5%2026.0618.411.59
Streamline (Layer)Model=Llama-3.2-3B, Sparsity=12.5%2026.0618.591.71
SUBFITModel=Llama-3.1-8B, Sparsity=25%2026.0619.32.06
Streamline (FFN)Model=Llama-3.2-3B, Sparsity=12.5%2026.0619.481.88
DenseModel=Qwen3-4B, Sparsity=0%2026.0619.891
SUBFITModel=Qwen3-8B, Sparsity=12.5%2026.0620.131.46
SUBFITModel=DeepSeek-7B, Sparsity=25%2026.0621.442.37
Streamline (Layer)Model=Qwen3-8B, Sparsity=12.5%2026.0622.781.51
SUBFITModel=Llama-3.2-3B, Sparsity=25%2026.0623.422.11
Streamline (FFN)Model=Qwen3-8B, Sparsity=12.5%2026.0623.841.69
SUBFITModel=Qwen3-4B, Sparsity=12.5%2026.0623.871.27
ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=12.5%2026.0624.831.73
Streamline (Layer)Model=Llama-3.1-8B, Sparsity=25%2026.0624.842.5
Streamline (FFN)Model=Llama-3.1-8B, Sparsity=25%2026.0625.62.64
ReplaceMe (LS)Model=Qwen3-8B, Sparsity=12.5%2026.0625.961.87
ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=12.5%2026.0628.561.43
Streamline (FFN)Model=Qwen3-4B, Sparsity=12.5%2026.0628.831.55
Streamline (Layer)Model=DeepSeek-7B, Sparsity=25%2026.0629.383.05
ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=25%2026.0629.643.11
ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=25%2026.0629.843.42
ReplaceMe (LS)Model=Qwen3-4B, Sparsity=12.5%2026.0630.861.59
ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=25%2026.0631.342.77
Streamline (Layer)Model=Llama-3.2-3B, Sparsity=25%2026.0633.623.28
ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=25%2026.0634.63.85
ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=25%2026.0635.283.98
ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=25%2026.0636.353.63
Streamline (FFN)Model=Llama-3.2-3B, Sparsity=25%2026.0636.683.59
Streamline (FFN)Model=DeepSeek-7B, Sparsity=25%2026.0638.774.17
SUBFITModel=Qwen3-8B, Sparsity=25%2026.0640.063.18
SUBFITModel=Qwen3-4B, Sparsity=25%2026.0643.522.54
ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=25%2026.0677.966.8
ReplaceMe (LS)Model=Qwen3-8B, Sparsity=25%2026.0689.457.77
ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=25%2026.0693.015.37
ReplaceMe (LS)Model=Qwen3-4B, Sparsity=25%2026.06120.57.38
Streamline (Layer)Model=Qwen3-8B, Sparsity=25%2026.06204.4321.92
Streamline (Layer)Model=Qwen3-4B, Sparsity=25%2026.06282.5523.04
Streamline (FFN)Model=Qwen3-4B, Sparsity=25%2026.06329.1328.57
Streamline (FFN)Model=Qwen3-8B, Sparsity=25%2026.06348.0735.14