Language Modeling on WikiText (Best CE, PPL)
1.32Best CEEnd-to-end Transformer ref.
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| End-to-end Transformer ref.Setup=same byte setup, Training=full-model training2026.06 | 1.32 | 3.74 | |
| BlockTrain long runBlocks=3, Layers per block=4, Rounds=322026.06 | 1.359 | 3.89 | |
| Exact Transformer, fullBlocks=4, Layers per block=3, Rounds=162026.06 | 1.566 | 4.79 | |
| Exact Transformer, shortBlocks=4, Layers per block=3, Run length=short run2026.06 | 2.592 | 13.4 | |
| Exact MLP blockNotes=exact mechanics, weak block2026.06 | 2.74 | 15.5 | |
| Direct embedding-MSE denoiserNotes=approximation, not exact mechanics2026.06 | 3.243 | 25.6 |