Unconditional generation on LM1B
36.42Generation PerplexityAR (student-size)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AR (student-size)Training Steps=1M, Sequence Length=128, Reference Type=Autoregressive2026.03 | 36.42 | — | 0.465 | |
| BD3-LM + Gumbel DistillationTraining Steps=1M, Sequence Length=128, L'=4, Teacher=GPT-2-Large2026.03 | 46.06 | — | 0.291 | |
| IMDMDistillation Method=SDTT + ReDi, Sampling Steps=162026.05 | 52.51 | — | — | |
| MDLMDistillation Method=SDTT + ReDi, Sampling Steps=162026.05 | 55.36 | — | — | |
| BD3-LMTraining Steps=1M, Sequence Length=128, L'=42026.03 | 56.98 | — | 0.193 | |
| IMDMDistillation Method=SDTT + ReDi, Sampling Steps=82026.05 | 65.55 | — | — | |
| MDLM + Gumbel DistillationTraining Steps=1M, Sequence Length=128, Teacher=GPT-2-Large2026.03 | 67.64 | — | 0.264 | |
| MDLMDistillation Method=SDTT + ReDi, Sampling Steps=82026.05 | 72.04 | — | — | |
| MDLMTraining Steps=1M, Sequence Length=1282026.03 | 78.74 | — | 0.179 | |
| IMDMDistillation Method=ReDi, Sampling Steps=162026.05 | 81.57 | — | — | |
| MDLMDistillation Method=ReDi, Sampling Steps=162026.05 | 85.27 | — | — | |
| IMDMDistillation Method=SDTT, Sampling Steps=162026.05 | 90.16 | — | — | |
| MDLMDistillation Method=SDTT, Sampling Steps=162026.05 | 93.29 | — | — | |
| IMDMDistillation Method=SDTT + ReDi, Sampling Steps=42026.05 | 93.44 | — | — | |
| CoDARTemperature=12026.03 | 104.76 | 0.3264 | — | |
| IMDMDistillation Method=ReDi, Sampling Steps=82026.05 | 104.93 | — | — | |
| MDLMDistillation Method=ReDi, Sampling Steps=82026.05 | 113.14 | — | — | |
| IMDMDistillation Method=SDTT, Sampling Steps=82026.05 | 122.52 | — | — | |
| MDLMDistillation Method=SDTT + ReDi, Sampling Steps=42026.05 | 122.56 | — | — | |
| MDLMDistillation Method=SDTT, Sampling Steps=82026.05 | 126.07 | — | — | |
| IMDMDistillation Method=ReDi, Sampling Steps=42026.05 | 151.89 | — | — | |
| IMDMDistillation Method=SDTT + ReDi, Sampling Steps=22026.05 | 165.85 | — | — | |
| LD4LGTemperature=12026.03 | 167.47 | 0.5797 | — | |
| MDLMDistillation Method=ReDi, Sampling Steps=42026.05 | 195.39 | — | — | |
| IMDMDistillation Method=SDTT, Sampling Steps=42026.05 | 210.05 | — | — | |
| MDLMDistillation Method=SDTT, Sampling Steps=42026.05 | 227.08 | — | — | |
| IMDMDistillation Method=ReDi, Sampling Steps=22026.05 | 284.62 | — | — | |
| MDLMDistillation Method=SDTT + ReDi, Sampling Steps=22026.05 | 353.9 | — | — | |
| IMDMDistillation Method=SDTT, Sampling Steps=22026.05 | 524.31 | — | — | |
| MDLMDistillation Method=ReDi, Sampling Steps=22026.05 | 530.36 | — | — | |
| MDLMDistillation Method=SDTT, Sampling Steps=22026.05 | 584.22 | — | — |