Language Modeling on Pubmed (val)
36.5PerplexityLoopMDM
Evaluation results
| Method | Links | |
|---|---|---|
| LoopMDMS (inference-time loop count)=12, Zero-shot protocol=true, Model parameters=170M2026.05 | 36.5 | |
| LoopMDMS (inference-time loop count)=6, Zero-shot protocol=true, Model parameters=170M2026.05 | 37.2 | |
| DCDMParam=0.1B, Training Tokens=128B, Evaluation Protocol=Zero-shot2026.05 | 40.17 | |
| BDLMParam=0.1B, Training Tokens=256B, Evaluation Protocol=Zero-shot2026.05 | 42.52 | |
| MDMZero-shot protocol=true, Model parameters=170M2026.05 | 43 | |
| MDLMParam=0.1B, Training Tokens=256B, Evaluation Protocol=Zero-shot2026.05 | 43.13 | |
| BD3-LMTraining steps=250K, Training Dataset=OWT, Zero-shot=true, L'=162025.06 | 43.41 | |
| MDLMTraining steps=250K, Training Dataset=OWT, Zero-shot=true2025.06 | 46.51 | |
| SEDD AbsorbTraining steps=250K, Training Dataset=OWT, Zero-shot=true2025.06 | 47.07 | |
| LoopMDMS (inference-time loop count)=1, Zero-shot protocol=true, Model parameters=170M2026.05 | 47.9 | |
| ARParam=0.1B, Evaluation Protocol=Zero-shot2026.05 | 48.59 | |
| AR TransformerTraining steps=250K, Training Dataset=OWT, Zero-shot=true2025.06 | 49.49 | |
| Eso-LMsTraining steps=250K, Training Dataset=OWT, Zero-shot=true, alpha_0=0.52025.06 | 60.2 | |
| Eso-LMsTraining steps=250K, Training Dataset=OWT, Zero-shot=true, alpha_0=12025.06 | 62.37 | |
| Eso-LMsTraining steps=250K, Training Dataset=OWT, Zero-shot=true, alpha_0=0.252025.06 | 62.45 | |
| Eso-LMsTraining steps=250K, Training Dataset=OWT, Zero-shot=true, alpha_0=0.1252025.06 | 65.27 |