Character-level Language Modeling on enwik8 one-million-parameter scale (val)
1.5953BPCFSN, converged
Evaluation Results
| Method | Links | |
|---|---|---|
| FSN, convergedEpochs=50, Number of parameters=1M2026.06 | 1.5953 | |
| FSN, no phasesEpochs=30, Number of parameters=1M2026.06 | 1.6031 | |
| FSNEpochs=30, Number of parameters=1M2026.06 | 1.605 | |
| Transformer, convergedEpochs=50, Number of parameters=1M2026.06 | 1.611 | |
| Transformer (RoPE, SwiGLU)Epochs=30, Number of parameters=1M2026.06 | 1.6258 | |
| FSN-MF, winding registerEpochs=30, Number of parameters=1M2026.06 | 1.6405 | |
| FSN-MF (no MLP anywhere; n=2)Epochs=30, Number of parameters=1M2026.06 | 1.6452 |