Language Modeling on Lambada (Perplexity)
14.36Perplexity (Lambada)Dense
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| DenseModel=DeepSeek-7B, Sparsity=0%2026.06 | 14.36 | — | 1 | — | |
| ParcaeNumber of Parameters=1.3B, T=82026.04 | 14.71 | — | — | — | |
| TransformerNumber of Parameters=1.3B, T=-2026.04 | 17.26 | — | — | — | |
| DenseModel=Llama-3.1-8B, Sparsity=0%2026.06 | 17.78 | — | 1 | — | |
| ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 18.21 | — | 1.42 | — | |
| ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 18.75 | — | 1.46 | — | |
| Streamline (Layer)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 19.33 | — | 1.46 | — | |
| ParcaeNumber of Parameters=770M, T=82026.04 | 19.71 | — | — | — | |
| DenseModel=Llama-3.2-3B, Sparsity=0%2026.06 | 20.15 | — | 1 | — | |
| Streamline (FFN)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 20.37 | — | 1.58 | — | |
| Streamline (Layer)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 21.65 | — | 1.41 | — | |
| Streamline (FFN)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 22.1 | — | 1.44 | — | |
| ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 22.14 | — | 1.42 | — | |
| ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 22.35 | — | 1.43 | — | |
| TransformerNumber of Parameters=770M, T=-2026.04 | 22.37 | — | — | — | |
| SUBFITModel=DeepSeek-7B, Sparsity=12.5%2026.06 | 23.17 | — | 1.52 | — | |
| SUBFITModel=Llama-3.1-8B, Sparsity=12.5%2026.06 | 24.24 | — | 1.4 | — | |
| DenseModel=Qwen3-8B, Sparsity=0%2026.06 | 25.7 | — | 1 | — | |
| SUBFITModel=Llama-3.2-3B, Sparsity=12.5%2026.06 | 27.25 | — | 1.39 | — | |
| ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 27.84 | — | 1.57 | — | |
| Streamline (Layer)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 27.92 | — | 1.71 | — | |
| ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 28.35 | — | 1.59 | — | |
| SUBFITModel=DeepSeek-7B, Sparsity=25%2026.06 | 31.35 | — | 2.37 | — | |
| Streamline (FFN)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 31.65 | — | 1.88 | — | |
| ParcaeNumber of Parameters=370M, T=82026.04 | 32.74 | — | — | — | |
| SUBFITModel=Llama-3.1-8B, Sparsity=25%2026.06 | 33.09 | — | 2.06 | — | |
| DenseModel=Qwen3-4B, Sparsity=0%2026.06 | 33.8 | — | 1 | — | |
| Streamline (Layer)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 33.88 | — | 2.5 | — | |
| SUBFITModel=Qwen3-8B, Sparsity=12.5%2026.06 | 34.32 | — | 1.46 | — | |
| ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=25%2026.06 | 34.35 | — | 3.11 | — | |
| Manta-LMZero-shot=true, Protocol=ELBO-based proxy, Model parameters=101M2026.05 | 34.8 | — | — | — | |
| Streamline (Layer)Model=DeepSeek-7B, Sparsity=25%2026.06 | 35.37 | — | 3.05 | — | |
| Streamline (FFN)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 35.9 | — | 2.64 | — | |
| Streamline (Layer)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 36.3 | — | 1.51 | — | |
| ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 37.05 | — | 1.73 | — | |
| Streamline (FFN)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 37.52 | — | 1.69 | — | |
| ReplaceMe (LS)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 38.69 | — | 1.87 | — | |
| SUBFITModel=Llama-3.2-3B, Sparsity=25%2026.06 | 39.17 | — | 2.11 | — | |
| SUBFITModel=Qwen3-4B, Sparsity=12.5%2026.06 | 39.4 | — | 1.27 | — | |
| TransformerNumber of Parameters=370M, T=-2026.04 | 40.77 | — | — | — | |
| ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=25%2026.06 | 41.79 | — | 3.98 | — | |
| ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 43.28 | — | 3.42 | — | |
| GPT-2Zero-shot=true2026.05 | 45.04 | — | — | — | |
| ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 45.4 | — | 1.43 | — | |
| Streamline (FFN)Model=DeepSeek-7B, Sparsity=25%2026.06 | 45.88 | — | 4.17 | — | |
| Streamline (FFN)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 46.78 | — | 1.55 | — | |
| Streamline (Layer)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 47.93 | — | 3.28 | — | |
| MDLMZero-shot=true2026.06 | 48.29 | — | — | — | |
| MD4Zero-shot=true, Protocol=ELBO-based proxy2026.05 | 48.43 | — | — | — | |
| ReplaceMe (LS)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 48.74 | — | 1.59 | — | |
| SEDDZero-shot=true2026.06 | 48.93 | — | — | — | |
| BD3LMZero-shot=true, Block Size (BS)=42026.06 | 50.03 | — | — | — | |
| ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 50.22 | — | 2.77 | — | |
| SEDDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 50.92 | — | — | — | |
| ABDZero-shot=true, Block Size (BS)=12026.06 | 51 | — | — | — | |
| Streamline (FFN)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 51.22 | — | 3.59 | — | |
| ABDZero-shot=true, Block Size (BS)=42026.06 | 51.38 | — | — | — | |
| RADDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 51.7 | — | — | — | |
| ARZero-shot=true2026.06 | 52.13 | — | — | — | |
| ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 52.15 | — | 3.63 | — | |
| ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 54.63 | — | 3.85 | — | |
| AO-GPTZero-shot=true2026.05 | 54.92 | — | — | — | |
| PLAIDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 57.28 | — | — | — | |
| SUBFITModel=Qwen3-8B, Sparsity=25%2026.06 | 63.22 | — | 3.18 | — | |
| SUBFITModel=Qwen3-4B, Sparsity=25%2026.06 | 69.26 | — | 2.54 | — | |
| ParcaeNumber of Parameters=140M, T=82026.04 | 80.64 | — | — | — | |
| D3PMZero-shot=true, Protocol=ELBO-based proxy2026.05 | 93.47 | — | — | — | |
| ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=25%2026.06 | 102.44 | — | 6.8 | — | |
| MIRParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | 106.7115 | — | — | 22.61 | |
| ReplaceMe (LS)Model=Qwen3-8B, Sparsity=25%2026.06 | 112.81 | — | 7.77 | — | |
| Regularized BaselineParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | 112.8966 | — | — | 22.71 | |
| TransformerNumber of Parameters=140M, T=-2026.04 | 127.39 | — | — | — | |
| ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=25%2026.06 | 132.37 | — | 5.37 | — | |
| ReplaceMe (LS)Model=Qwen3-4B, Sparsity=25%2026.06 | 151.88 | — | 7.38 | — | |
| Streamline (Layer)Model=Qwen3-8B, Sparsity=25%2026.06 | 229.51 | — | 21.92 | — | |
| Streamline (Layer)Model=Qwen3-4B, Sparsity=25%2026.06 | 314.53 | — | 23.04 | — | |
| Streamline (FFN)Model=Qwen3-8B, Sparsity=25%2026.06 | 356.48 | — | 35.14 | — | |
| Streamline (FFN)Model=Qwen3-4B, Sparsity=25%2026.06 | 448.64 | — | 28.57 | — | |
| Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 51.28 | — | — | |
| D3PM AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04 | — | 93.47 | — | — | |
| DenseBase Model=Qwen3-8B, Sparsity Level=Dense2026.06 | — | 25.7 | — | — | |
| DenseBase Model=DeepSeek-7B, Sparsity Level=Dense2026.06 | — | 14.36 | — | — | |
| DenseBase Model=Llama-3.2-3B, Sparsity Level=Dense2026.06 | — | 20.15 | — | — | |
| DenseBase Model=Qwen3-4B, Sparsity Level=Dense2026.06 | — | 33.8 | — | — | |
| DenseBase Model=Llama-3.1-8B, Sparsity Level=Dense2026.06 | — | 17.78 | — | — | |
| DenseBackbone=Qwen3-8B, Sparsity=0%2026.06 | — | 25.7 | — | — | |
| DenseBackbone=Llama-3.2-3B, Sparsity=0%2026.06 | — | 20.15 | — | — | |
| DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 49.78 | — | — | |
| LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 46.93 | — | — | |
| MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 47.52 | — | — | |
| MDLMZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | — | 50.04 | — | — | |
| PlaidModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04 | — | 57.28 | — | — | |
| Random GuessParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | — | — | — | 0 | |
| ReplaceMe (Cosine)Base Model=Qwen3-8B, Sparsity Level=37.5%2026.06 | — | 124.01 | — | — | |
| ReplaceMe (Cosine)Base Model=Qwen3-4B, Sparsity Level=37.5%2026.06 | — | 5,394.41 | — | — | |
| SCMDMAddZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | — | 49.1 | — | — | |
| SCMDMConcatZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | — | 49.09 | — | — | |
| SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 49.86 | — | — | |
| SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | — | 57.29 | — | — | |
| Streamline (FFN)Base Model=Qwen3-8B, Sparsity Level=20%2026.06 | — | 45.75 | — | — |