Language Modeling on OpenWebText (OWT) (PPL)
11.2PPLOriginal Model
Evaluation results
| Method | Links | |
|---|---|---|
| Original ModelBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=0% / 0%, Mem (GB)=5.80, Time (s)=41002026.06 | 11.2 | |
| Learning to AllocateBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=38% / 28%, Mem (GB)=5.30, Time (s)=28502026.06 | 11.32 | |
| Oracle Static PruningBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=33% / 22%, Mem (GB)=4.95, Time (s)=26802026.06 | 11.35 | |
| FlexiDepthBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=34% / –, Mem (GB)=5.75, Time (s)=29102026.06 | 11.38 | |
| AdaSkipBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=36% / –, Mem (GB)=5.65, Time (s)=28002026.06 | 11.42 | |
| Static PruningBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=33% / 0%, Mem (GB)=5.10, Time (s)=27502026.06 | 12.95 |