Language Modeling on AG News
4.76PPLPruning
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PruningBackbone=Llama-2-7b2026.02 | 4.76 | 1.56 | |
| OriginalBackbone=Llama-2-7b2026.02 | 4.79 | 1.57 | |
| PruningBackbone=Llama-3-8b2026.02 | 5.69 | 1.74 | |
| PruningBackbone=Llama-3.1-8b2026.02 | 5.7 | 1.74 | |
| OriginalBackbone=Llama-3-8b2026.02 | 5.74 | 1.75 | |
| OriginalBackbone=Llama-3.1-8b2026.02 | 5.75 | 1.75 | |
| GhostBackbone=Llama-2-7b2026.02 | 6.95 | 1.94 | |
| GhostBackbone=Llama-3-8b2026.02 | 7.65 | 2.04 | |
| GhostBackbone=Llama-3.1-8b2026.02 | 7.75 | 2.05 | |
| PruningBackbone=GPT-22026.02 | 8.9 | 2.91 | |
| OriginalBackbone=GPT-22026.02 | 9.07 | 2.2 | |
| GhostBackbone=GPT-22026.02 | 12.91 | 2.56 | |
| CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 27.67 | — | |
| ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 30.62 | — | |
| BD3LMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 41.18 | — | |
| Llama3Method=Exact2026.03 | 41.29 | — | |
| MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 42.2 | — | |
| DID-FSize=Medium, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 48.84 | — | |
| RADDSize=Medium, Zero-shot=true2026.03 | 48.96 | — | |
| Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 52.09 | — | |
| ARZero-shot=true2026.06 | 52.11 | — | |
| DID-SSize=Medium, Zero-shot=true, Alignment=Steps-aligned2026.03 | 52.53 | — | |
| GIDDSetting=Zero-shot2026.02 | 60.607 | — | |
| MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 61.15 | — | |
| ABDZero-shot=true, Block Size (BS)=12026.06 | 61.25 | — | |
| MDLMSetting=Zero-shot2026.02 | 61.374 | — | |
| BD3LMZero-shot=true, Block Size (BS)=42026.06 | 61.67 | — | |
| SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 62.09 | — | |
| ABDZero-shot=true, Block Size (BS)=42026.06 | 62.46 | — | |
| XDLMSetting=Zero-shot2026.02 | 62.768 | — | |
| MDLMZero-shot=true2026.06 | 62.78 | — | |
| DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 67.81 | — | |
| SEDDZero-shot=true2026.06 | 67.82 | — | |
| SCMDMAddZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 68.03 | — | |
| SCMDMConcatZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 68.13 | — | |
| UDLMSetting=Zero-shot2026.02 | 69.402 | — | |
| LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 69.41 | — | |
| MDLMZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.04 | 70.6 | — | |
| DID-FSize=Small, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 71.48 | — | |
| RADDSize=Small, Zero-shot=true2026.03 | 73.18 | — | |
| DID-SSize=Small, Zero-shot=true, Alignment=Steps-aligned2026.03 | 76.02 | — | |
| LLaDAMethod=DUEL2026.03 | 78.91 | — | |
| UDLMModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 80.96 | — | |
| SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 82.64 | — | |
| LLaDAMethod=ELBO2026.03 | 85.17 | — |