List your company
ResearchFormerly WizWand

Language Modeling on Lambada (Perplexity)

14.36Perplexity (Lambada)Dense
-3.0112114.2444231.5348.7556Apr 14, 2026Apr 26, 2026May 9, 2026May 21, 2026Jun 3, 2026Jun 15, 2026Jun 28, 2026
Updated 4mo ago

Evaluation results

MethodLinks
DenseModel=DeepSeek-7B, Sparsity=0%2026.0614.36—1—
ParcaeNumber of Parameters=1.3B, T=82026.0414.71———
TransformerNumber of Parameters=1.3B, T=-2026.0417.26———
DenseModel=Llama-3.1-8B, Sparsity=0%2026.0617.78—1—
ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=12.5%2026.0618.21—1.42—
ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=12.5%2026.0618.75—1.46—
Streamline (Layer)Model=DeepSeek-7B, Sparsity=12.5%2026.0619.33—1.46—
ParcaeNumber of Parameters=770M, T=82026.0419.71———
DenseModel=Llama-3.2-3B, Sparsity=0%2026.0620.15—1—
Streamline (FFN)Model=DeepSeek-7B, Sparsity=12.5%2026.0620.37—1.58—
Streamline (Layer)Model=Llama-3.1-8B, Sparsity=12.5%2026.0621.65—1.41—
Streamline (FFN)Model=Llama-3.1-8B, Sparsity=12.5%2026.0622.1—1.44—
ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=12.5%2026.0622.14—1.42—
ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=12.5%2026.0622.35—1.43—
TransformerNumber of Parameters=770M, T=-2026.0422.37———
SUBFITModel=DeepSeek-7B, Sparsity=12.5%2026.0623.17—1.52—
SUBFITModel=Llama-3.1-8B, Sparsity=12.5%2026.0624.24—1.4—
DenseModel=Qwen3-8B, Sparsity=0%2026.0625.7—1—
SUBFITModel=Llama-3.2-3B, Sparsity=12.5%2026.0627.25—1.39—
ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=12.5%2026.0627.84—1.57—
Streamline (Layer)Model=Llama-3.2-3B, Sparsity=12.5%2026.0627.92—1.71—
ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=12.5%2026.0628.35—1.59—
SUBFITModel=DeepSeek-7B, Sparsity=25%2026.0631.35—2.37—
Streamline (FFN)Model=Llama-3.2-3B, Sparsity=12.5%2026.0631.65—1.88—
ParcaeNumber of Parameters=370M, T=82026.0432.74———
SUBFITModel=Llama-3.1-8B, Sparsity=25%2026.0633.09—2.06—
DenseModel=Qwen3-4B, Sparsity=0%2026.0633.8—1—
Streamline (Layer)Model=Llama-3.1-8B, Sparsity=25%2026.0633.88—2.5—
SUBFITModel=Qwen3-8B, Sparsity=12.5%2026.0634.32—1.46—
ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=25%2026.0634.35—3.11—
Manta-LMZero-shot=true, Protocol=ELBO-based proxy, Model parameters=101M2026.0534.8———
Streamline (Layer)Model=DeepSeek-7B, Sparsity=25%2026.0635.37—3.05—
Streamline (FFN)Model=Llama-3.1-8B, Sparsity=25%2026.0635.9—2.64—
Streamline (Layer)Model=Qwen3-8B, Sparsity=12.5%2026.0636.3—1.51—
ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=12.5%2026.0637.05—1.73—
Streamline (FFN)Model=Qwen3-8B, Sparsity=12.5%2026.0637.52—1.69—
ReplaceMe (LS)Model=Qwen3-8B, Sparsity=12.5%2026.0638.69—1.87—
SUBFITModel=Llama-3.2-3B, Sparsity=25%2026.0639.17—2.11—
SUBFITModel=Qwen3-4B, Sparsity=12.5%2026.0639.4—1.27—
TransformerNumber of Parameters=370M, T=-2026.0440.77———
ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=25%2026.0641.79—3.98—
ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=25%2026.0643.28—3.42—
GPT-2Zero-shot=true2026.0545.04———
ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=12.5%2026.0645.4—1.43—
Streamline (FFN)Model=DeepSeek-7B, Sparsity=25%2026.0645.88—4.17—
Streamline (FFN)Model=Qwen3-4B, Sparsity=12.5%2026.0646.78—1.55—
Streamline (Layer)Model=Llama-3.2-3B, Sparsity=25%2026.0647.93—3.28—
MDLMZero-shot=true2026.0648.29———
MD4Zero-shot=true, Protocol=ELBO-based proxy2026.0548.43———
ReplaceMe (LS)Model=Qwen3-4B, Sparsity=12.5%2026.0648.74—1.59—
SEDDZero-shot=true2026.0648.93———
BD3LMZero-shot=true, Block Size (BS)=42026.0650.03———
ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=25%2026.0650.22—2.77—
SEDDZero-shot=true, Protocol=ELBO-based proxy2026.0550.92———
ABDZero-shot=true, Block Size (BS)=12026.0651———
Streamline (FFN)Model=Llama-3.2-3B, Sparsity=25%2026.0651.22—3.59—
ABDZero-shot=true, Block Size (BS)=42026.0651.38———
RADDZero-shot=true, Protocol=ELBO-based proxy2026.0551.7———
ARZero-shot=true2026.0652.13———
ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=25%2026.0652.15—3.63—
ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=25%2026.0654.63—3.85—
AO-GPTZero-shot=true2026.0554.92———
PLAIDZero-shot=true, Protocol=ELBO-based proxy2026.0557.28———
SUBFITModel=Qwen3-8B, Sparsity=25%2026.0663.22—3.18—
SUBFITModel=Qwen3-4B, Sparsity=25%2026.0669.26—2.54—
ParcaeNumber of Parameters=140M, T=82026.0480.64———
D3PMZero-shot=true, Protocol=ELBO-based proxy2026.0593.47———
ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=25%2026.06102.44—6.8—
MIRParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06106.7115——22.61
ReplaceMe (LS)Model=Qwen3-8B, Sparsity=25%2026.06112.81—7.77—
Regularized BaselineParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06112.8966——22.71
TransformerNumber of Parameters=140M, T=-2026.04127.39———
ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=25%2026.06132.37—5.37—
ReplaceMe (LS)Model=Qwen3-4B, Sparsity=25%2026.06151.88—7.38—
Streamline (Layer)Model=Qwen3-8B, Sparsity=25%2026.06229.51—21.92—
Streamline (Layer)Model=Qwen3-4B, Sparsity=25%2026.06314.53—23.04—
Streamline (FFN)Model=Qwen3-8B, Sparsity=25%2026.06356.48—35.14—
Streamline (FFN)Model=Qwen3-4B, Sparsity=25%2026.06448.64—28.57—
Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—51.28——
D3PM AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04—93.47——
DenseBase Model=Qwen3-8B, Sparsity Level=Dense2026.06—25.7——
DenseBase Model=DeepSeek-7B, Sparsity Level=Dense2026.06—14.36——
DenseBase Model=Llama-3.2-3B, Sparsity Level=Dense2026.06—20.15——
DenseBase Model=Qwen3-4B, Sparsity Level=Dense2026.06—33.8——
DenseBase Model=Llama-3.1-8B, Sparsity Level=Dense2026.06—17.78——
DenseBackbone=Qwen3-8B, Sparsity=0%2026.06—25.7——
DenseBackbone=Llama-3.2-3B, Sparsity=0%2026.06—20.15——
DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—49.78——
LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—46.93——
MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—47.52——
MDLMZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04—50.04——
PlaidModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04—57.28——
Random GuessParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06———0
ReplaceMe (Cosine)Base Model=Qwen3-8B, Sparsity Level=37.5%2026.06—124.01——
ReplaceMe (Cosine)Base Model=Qwen3-4B, Sparsity Level=37.5%2026.06—5,394.41——
SCMDMAddZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04—49.1——
SCMDMConcatZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04—49.09——
SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—49.86——
SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04—57.29——
Streamline (FFN)Base Model=Qwen3-8B, Sparsity Level=20%2026.06—45.75——