Language Modeling on LAMBADA (test)
4PerplexityLLaMA2-7B
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| LLaMA2-7BNumber of parameters=7B2024.07 | 4 | — | — | |
| Qwen3 8B2026.02 | 4.6 | — | — | |
| SE-KD2026.02 | 6.9 | — | — | |
| SE-KD + TopSmpselection ratio=20%2026.02 | 6.9 | — | — | |
| LLM-Pruner-2.6BNumber of parameters=2.6B, Model Architecture=TransAct2024.07 | 7.1 | — | — | |
| Full KD2026.02 | 7.3 | — | — | |
| SE-KD3X2026.02 | 7.3 | — | — | |
| TransAct-2.6BNumber of parameters=2.6B2024.07 | 7.4 | — | — | |
| AT-KD2026.02 | 7.4 | — | — | |
| RS-KD2026.02 | 7.4 | — | — | |
| TopSmpselection ratio=20%2026.02 | 7.4 | — | — | |
| TopSmp + RS-KDselection ratio=20%2026.02 | 7.4 | — | — | |
| Sheared-LLaMA-2.7BNumber of parameters=2.7B2024.07 | 7.5 | — | — | |
| RandomPosselection ratio=20%2026.02 | 7.6 | — | — | |
| TOPSize=7B, Zero-shot=true2025.08 | 7.64 | 57.03 | — | |
| NTPSize=7B, Zero-shot=true2025.08 | 7.97 | 55.89 | — | |
| RandomSmpselection ratio=20%2026.02 | 8.2 | — | — | |
| DS-MTPSize=7B, Zero-shot=true2025.08 | 8.52 | 55.62 | — | |
| MTPSize=7B, Zero-shot=true2025.08 | 8.99 | 53.13 | — | |
| TOPSize=1.8B, Zero-shot=true2025.08 | 11.19 | 50.34 | — | |
| NTPSize=1.8B, Zero-shot=true2025.08 | 11.38 | 49.58 | — | |
| Sheared-LLaMA-1.3BNumber of parameters=1.3B2024.07 | 11.4 | — | — | |
| Qwen3 1.7B2026.02 | 12.2 | — | — | |
| TransAct-1.3BNumber of parameters=1.3B2024.07 | 13 | — | — | |
| LLM-Pruner-1.3BNumber of parameters=1.3B, Model Architecture=TransAct2024.07 | 13.2 | — | — | |
| DS-MTPSize=1.8B, Zero-shot=true2025.08 | 13.32 | 48.71 | — | |
| MTPSize=1.8B, Zero-shot=true2025.08 | 13.69 | 47.93 | — | |
| DenseSparsity=0%, Backbone=Llama-3.1-8B2026.06 | 17.78 | — | — | |
| DenseSparsity=0%, Backbone=Llama-3.2-3B2026.06 | 20.15 | — | — | |
| DIVE 1/8FFN Size=50% x 12025.06 | 24.84 | — | — | |
| Streamline (Layer)Sparsity=20%, Backbone=Llama-3.1-8B2026.06 | 25.79 | — | — | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.1-8B2026.06 | 26.59 | — | — | |
| SUBFITSparsity=20%, Backbone=Llama-3.1-8B2026.06 | 28.04 | — | — | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.1-8B2026.06 | 28.16 | — | — | |
| Streamline (FFN)Sparsity=20%, Backbone=Llama-3.1-8B2026.06 | 28.55 | — | — | |
| TOPSize=340M, Zero-shot=true2025.08 | 28.76 | 37.07 | — | |
| NTPSize=340M, Zero-shot=true2025.08 | 30.34 | 36.35 | — | |
| SUBFITSparsity=20%, Backbone=Llama-3.2-3B2026.06 | 31.94 | — | — | |
| FLAPFFN Size=50%2025.06 | 33.22 | — | — | |
| DenseSparsity=0%, Backbone=Qwen3-4B2026.06 | 33.8 | — | — | |
| MTPSize=340M, Zero-shot=true2025.08 | 35.31 | 35.32 | — | |
| GPT-2Size=Medium, Zero-shot=true, Unconditional=true2023.10 | 35.66 | — | — | |
| Streamline (Layer)Sparsity=20%, Backbone=Llama-3.2-3B2026.06 | 38.15 | — | — | |
| DS-MTPSize=340M, Zero-shot=true2025.08 | 40.99 | 34.66 | — | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.2-3B2026.06 | 42.61 | — | — | |
| Streamline (FFN)Sparsity=20%, Backbone=Llama-3.2-3B2026.06 | 43.68 | — | — | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.2-3B2026.06 | 43.74 | — | — | |
| SUBFITSparsity=30%, Backbone=Llama-3.1-8B2026.06 | 44.05 | — | — | |
| LDDM-MDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.10 | 44.22 | — | — | |
| GPT-2Size=Small, Zero-shot=true, Unconditional=true2023.10 | 45.04 | — | — | |
| EDLM-NCEzero-shot=true2024.10 | 46.92 | — | — | |
| VADDZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 47.3 | — | — | |
| MLDMzero-shot=true2024.10 | 47.52 | — | — | |
| MDLMDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 48.36 | — | — | |
| RADD-AOZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 49.43 | — | — | |
| MDLMZero-shot=true, Trained on=OpenWebText, Training iterations=1M, Reproduced=true2025.05 | 49.67 | — | — | |
| SUBFITSparsity=30%, Backbone=Llama-3.2-3B2026.06 | 49.67 | — | — | |
| EDLM-ARzero-shot=true2024.10 | 49.7 | — | — | |
| SEDDzero-shot=true2024.10 | 49.86 | — | — | |
| Streamline (Layer)Sparsity=30%, Backbone=Llama-3.1-8B2026.06 | 49.86 | — | — | |
| EDLM-coARzero-shot=true2024.10 | 50.04 | — | — | |
| SEDD AbsorbDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 50.15 | — | — | |
| SEDD-AbsorbZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 50.92 | — | — | |
| ARzero-shot=true2024.10 | 51.28 | — | — | |
| GPT-2Zero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 51.28 | — | — | |
| UDLMDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 51.68 | — | — | |
| DIVE 2/8FFN Size=25% x 22025.06 | 51.95 | — | — | |
| LDDM-UDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.10 | 52.34 | — | — | |
| SUBFITSparsity=20%, Backbone=Qwen3-4B2026.06 | 54.36 | — | — | |
| Streamline (Layer)Sparsity=30%, Backbone=Llama-3.2-3B2026.06 | 55.31 | — | — | |
| LLM-PrunerFFN Size=50%2025.06 | 56.66 | — | — | |
| Streamline (FFN)Sparsity=30%, Backbone=Llama-3.1-8B2026.06 | 58.06 | — | — | |
| Streamline (FFN)Sparsity=30%, Backbone=Llama-3.2-3B2026.06 | 65.94 | — | — | |
| ReplaceMe (LS)Sparsity=30%, Backbone=Llama-3.2-3B2026.06 | 66.21 | — | — | |
| ReplaceMe (Cosine)Sparsity=30%, Backbone=Llama-3.2-3B2026.06 | 67.11 | — | — | |
| ReplaceMe (LS)Sparsity=30%, Backbone=Llama-3.1-8B2026.06 | 68.58 | — | — | |
| SUBFITSparsity=37.5%, Backbone=Llama-3.1-8B2026.06 | 69.11 | — | — | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Qwen3-4B2026.06 | 74.98 | — | — | |
| SUBFITSparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 75.27 | — | — | |
| Streamline (Layer)Sparsity=20%, Backbone=Qwen3-4B2026.06 | 78.93 | — | — | |
| Streamline (Layer)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.06 | 81.19 | — | — | |
| LLaMA-MoE 1/8FFN Size=50% x 12025.06 | 87.27 | — | — | |
| SUBFITSparsity=30%, Backbone=Qwen3-4B2026.06 | 92.19 | — | — | |
| ReplaceMe (LS)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 94.43 | — | — | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Qwen3-4B2026.06 | 94.79 | — | — | |
| Streamline (FFN)Sparsity=20%, Backbone=Qwen3-4B2026.06 | 95.64 | — | — | |
| Streamline (FFN)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.06 | 98.14 | — | — | |
| ReplaceMe (Cosine)Sparsity=30%, Backbone=Llama-3.1-8B2026.06 | 99.62 | — | — | |
| ReplaceMe (Cosine)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 108.02 | — | — | |
| Streamline (Layer)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 108.99 | — | — | |
| ReplaceMe (LS)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.06 | 109.84 | — | — | |
| LLaMA-MoE 2/8FFN Size=25% x 22025.06 | 121 | — | — | |
| SUBFITSparsity=37.5%, Backbone=Qwen3-4B2026.06 | 124.51 | — | — | |
| Streamline (FFN)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 129.14 | — | — | |
| Universal Transformerhalting=dynamic2018.07 | 142 | 19 | — | |
| Universal Transformersteps=8, halting=fixed2018.07 | 202 | 18 | — | |
| Universal Transformersteps=9, halting=fixed2018.07 | 239 | 17 | — | |
| ReplaceMe (LS)Sparsity=30%, Backbone=Qwen3-4B2026.06 | 261.67 | — | — | |
| Streamline (FFN)Sparsity=30%, Backbone=Qwen3-4B2026.06 | 273.13 | — | — | |
| Universal Transformersteps=6, halting=fixed2018.07 | 319 | 17 | — |