Language Modeling on FineWeb 100M token (val)
12.11PerplexitySpectron
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SpectronModel Architecture=Factorized Transformer-L, Number of Parameters=454M2026.02 | 12.11 | — | — | |
| Self guidedModel Architecture=Factorized Transformer-L, Number of Parameters=454M2026.02 | 13.7 | — | — | |
| Naive (AdamW)Model Architecture=Factorized Transformer-L, Number of Parameters=454M2026.02 | 14.57 | — | — | |
| SpectronModel Architecture=Factorized Transformer-M, Number of Parameters=297M2026.02 | 14.62 | — | — | |
| Self guidedModel Architecture=Factorized Transformer-M, Number of Parameters=297M2026.02 | 15.53 | — | — | |
| Naive (AdamW)Model Architecture=Factorized Transformer-M, Number of Parameters=297M2026.02 | 15.54 | — | — | |
| SpectronModel Architecture=Factorized Transformer-S, Number of Parameters=94M2026.02 | 21.86 | — | — | |
| Self guidedModel Architecture=Factorized Transformer-S, Number of Parameters=94M2026.02 | 24.17 | — | — | |
| Naive (AdamW)Model Architecture=Factorized Transformer-S, Number of Parameters=94M2026.02 | 26.43 | — | — | |
| BaselineTotal Epochs=2562026.06 | — | 3.0476 | 10 | |
| q0Total Epochs=2562026.06 | — | 3.0034 | 12 | |
| q0 (scaled)Total Epochs=9602026.06 | — | 2.987 | 12.9 |