Text Generation on 1 Billion Words Dataset (LM1B) (test)
0.71MAUVEAR + DiffusionBlocks
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AR + DiffusionBlocksArchitecture=12-layer Llama-2-style, Blocks=B=4, Memory Reduction=4x2025.06 | 0.71 | 12.32 | 30.99 | |
| Huginn + DiffusionBlockstraining_mode=single-pass, iterations=02025.06 | 0.7 | 16.08 | 42.43 | |
| ARArchitecture=12-layer Llama-2-style2025.06 | 0.5 | 14.58 | 38.87 | |
| Huginntraining_mode=8-step truncated BPTT, iterations=322025.06 | 0.49 | 17.04 | 46.73 |