List your company
ResearchFormerly WizWand

Language Modeling on LAMBADA (test)

4PerplexityLLaMA2-7B
-8.676.45161.5246.55Jul 10, 2018Nov 2, 2019Feb 25, 2021Jun 20, 2022Oct 14, 2023Feb 5, 2025Jun 1, 2026
Updated 5mo ago

Evaluation results

MethodLinks
LLaMA2-7BNumber of parameters=7B2024.074——
Qwen3 8B2026.024.6——
SE-KD2026.026.9——
SE-KD + TopSmpselection ratio=20%2026.026.9——
LLM-Pruner-2.6BNumber of parameters=2.6B, Model Architecture=TransAct2024.077.1——
Full KD2026.027.3——
SE-KD3X2026.027.3——
TransAct-2.6BNumber of parameters=2.6B2024.077.4——
AT-KD2026.027.4——
RS-KD2026.027.4——
TopSmpselection ratio=20%2026.027.4——
TopSmp + RS-KDselection ratio=20%2026.027.4——
Sheared-LLaMA-2.7BNumber of parameters=2.7B2024.077.5——
RandomPosselection ratio=20%2026.027.6——
TOPSize=7B, Zero-shot=true2025.087.6457.03—
NTPSize=7B, Zero-shot=true2025.087.9755.89—
RandomSmpselection ratio=20%2026.028.2——
DS-MTPSize=7B, Zero-shot=true2025.088.5255.62—
MTPSize=7B, Zero-shot=true2025.088.9953.13—
TOPSize=1.8B, Zero-shot=true2025.0811.1950.34—
NTPSize=1.8B, Zero-shot=true2025.0811.3849.58—
Sheared-LLaMA-1.3BNumber of parameters=1.3B2024.0711.4——
Qwen3 1.7B2026.0212.2——
TransAct-1.3BNumber of parameters=1.3B2024.0713——
LLM-Pruner-1.3BNumber of parameters=1.3B, Model Architecture=TransAct2024.0713.2——
DS-MTPSize=1.8B, Zero-shot=true2025.0813.3248.71—
MTPSize=1.8B, Zero-shot=true2025.0813.6947.93—
DenseSparsity=0%, Backbone=Llama-3.1-8B2026.0617.78——
DenseSparsity=0%, Backbone=Llama-3.2-3B2026.0620.15——
DIVE 1/8FFN Size=50% x 12025.0624.84——
Streamline (Layer)Sparsity=20%, Backbone=Llama-3.1-8B2026.0625.79——
ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.1-8B2026.0626.59——
SUBFITSparsity=20%, Backbone=Llama-3.1-8B2026.0628.04——
ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.1-8B2026.0628.16——
Streamline (FFN)Sparsity=20%, Backbone=Llama-3.1-8B2026.0628.55——
TOPSize=340M, Zero-shot=true2025.0828.7637.07—
NTPSize=340M, Zero-shot=true2025.0830.3436.35—
SUBFITSparsity=20%, Backbone=Llama-3.2-3B2026.0631.94——
FLAPFFN Size=50%2025.0633.22——
DenseSparsity=0%, Backbone=Qwen3-4B2026.0633.8——
MTPSize=340M, Zero-shot=true2025.0835.3135.32—
GPT-2Size=Medium, Zero-shot=true, Unconditional=true2023.1035.66——
Streamline (Layer)Sparsity=20%, Backbone=Llama-3.2-3B2026.0638.15——
DS-MTPSize=340M, Zero-shot=true2025.0840.9934.66—
ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.2-3B2026.0642.61——
Streamline (FFN)Sparsity=20%, Backbone=Llama-3.2-3B2026.0643.68——
ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.2-3B2026.0643.74——
SUBFITSparsity=30%, Backbone=Llama-3.1-8B2026.0644.05——
LDDM-MDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.1044.22——
GPT-2Size=Small, Zero-shot=true, Unconditional=true2023.1045.04——
EDLM-NCEzero-shot=true2024.1046.92——
VADDZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.0547.3——
MLDMzero-shot=true2024.1047.52——
MDLMDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.1048.36——
RADD-AOZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.0549.43——
MDLMZero-shot=true, Trained on=OpenWebText, Training iterations=1M, Reproduced=true2025.0549.67——
SUBFITSparsity=30%, Backbone=Llama-3.2-3B2026.0649.67——
EDLM-ARzero-shot=true2024.1049.7——
SEDDzero-shot=true2024.1049.86——
Streamline (Layer)Sparsity=30%, Backbone=Llama-3.1-8B2026.0649.86——
EDLM-coARzero-shot=true2024.1050.04——
SEDD AbsorbDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.1050.15——
SEDD-AbsorbZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.0550.92——
ARzero-shot=true2024.1051.28——
GPT-2Zero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.0551.28——
UDLMDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.1051.68——
DIVE 2/8FFN Size=25% x 22025.0651.95——
LDDM-UDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.1052.34——
SUBFITSparsity=20%, Backbone=Qwen3-4B2026.0654.36——
Streamline (Layer)Sparsity=30%, Backbone=Llama-3.2-3B2026.0655.31——
LLM-PrunerFFN Size=50%2025.0656.66——
Streamline (FFN)Sparsity=30%, Backbone=Llama-3.1-8B2026.0658.06——
Streamline (FFN)Sparsity=30%, Backbone=Llama-3.2-3B2026.0665.94——
ReplaceMe (LS)Sparsity=30%, Backbone=Llama-3.2-3B2026.0666.21——
ReplaceMe (Cosine)Sparsity=30%, Backbone=Llama-3.2-3B2026.0667.11——
ReplaceMe (LS)Sparsity=30%, Backbone=Llama-3.1-8B2026.0668.58——
SUBFITSparsity=37.5%, Backbone=Llama-3.1-8B2026.0669.11——
ReplaceMe (Cosine)Sparsity=20%, Backbone=Qwen3-4B2026.0674.98——
SUBFITSparsity=37.5%, Backbone=Llama-3.2-3B2026.0675.27——
Streamline (Layer)Sparsity=20%, Backbone=Qwen3-4B2026.0678.93——
Streamline (Layer)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.0681.19——
LLaMA-MoE 1/8FFN Size=50% x 12025.0687.27——
SUBFITSparsity=30%, Backbone=Qwen3-4B2026.0692.19——
ReplaceMe (LS)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.0694.43——
ReplaceMe (LS)Sparsity=20%, Backbone=Qwen3-4B2026.0694.79——
Streamline (FFN)Sparsity=20%, Backbone=Qwen3-4B2026.0695.64——
Streamline (FFN)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.0698.14——
ReplaceMe (Cosine)Sparsity=30%, Backbone=Llama-3.1-8B2026.0699.62——
ReplaceMe (Cosine)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06108.02——
Streamline (Layer)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06108.99——
ReplaceMe (LS)Sparsity=37.5%, Backbone=Llama-3.1-8B2026.06109.84——
LLaMA-MoE 2/8FFN Size=25% x 22025.06121——
SUBFITSparsity=37.5%, Backbone=Qwen3-4B2026.06124.51——
Streamline (FFN)Sparsity=37.5%, Backbone=Llama-3.2-3B2026.06129.14——
Universal Transformerhalting=dynamic2018.0714219—
Universal Transformersteps=8, halting=fixed2018.0720218—
Universal Transformersteps=9, halting=fixed2018.0723917—
ReplaceMe (LS)Sparsity=30%, Backbone=Qwen3-4B2026.06261.67——
Streamline (FFN)Sparsity=30%, Backbone=Qwen3-4B2026.06273.13——
Universal Transformersteps=6, halting=fixed2018.0731917—