Character-level Language Modeling on codeparrot sequence length 256 (val)
1.2217Bits per Character (BPC)FSN, no phases
Evaluation Results
| Method | Links | |
|---|---|---|
| FSN, no phasesEpochs=30, Number of parameters=1M, Sequence length=2562026.06 | 1.2217 | |
| FSNEpochs=30, Number of parameters=1M, Sequence length=2562026.06 | 1.2235 | |
| Transformer (RoPE, SwiGLU)Epochs=30, Number of parameters=1M, Sequence length=2562026.06 | 1.2409 |