Language Modeling on ClimbMix pretraining corpus (val held-out)
0.7123BPB (val)MinGram
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MinGramTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7123 | — | 9 | |
| Unigram-BPE-InitTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7123 | 0 | 26 | |
| MinGram-PPTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7125 | -3 | 57 | |
| FSP-BPE-InitTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7126 | -4 | 98 | |
| UnigramTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7127 | -5 | 24 | |
| FSPTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7127 | -7 | 49 | |
| PathPiece-BPETokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7131 | -12 | 59 | |
| ConvexTokTokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7132 | -13 | 81 | |
| BPETokenizer training data=FineWeb-English, Model architecture=depth-24 nanochat, Training tokens=5.86B, Number of seeds=202026.06 | 0.7138 | -22 | 172 | |
| Bank of Values (BoV)Output=Ev[i], Layers=last 1/3, Coeff.=γv, Model Scale=780M, Budget=FLOP-controlled2026.06 | 0.714 | — | — | |
| V1 variant (last 1/3)Output=V1, Layers=last 1/3, Coeff.=γv, Model Scale=780M, Budget=FLOP-controlled2026.06 | 0.718 | — | — | |
| x0WV variantOutput=x0WV, Layers=last 1/3, Coeff.=γv, Model Scale=780M, Budget=FLOP-controlled2026.06 | 0.719 | — | — | |
| Standard AttentionOutput=V, Layers=—, Coeff.=—, Model Scale=780M, Budget=FLOP-controlled2026.06 | 0.722 | — | — | |
| V1 variant (every layer)Output=V1, Layers=every, Coeff.=1, Model Scale=780M, Budget=FLOP-controlled2026.06 | 0.741 | — | — |