Language Modeling on WikiText-103 word-level (val)
7.533PerplexityHarmonic
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HarmonicSeq. len=32,7682026.05 | 7.533 | 7.2 | |
| HarmonicSeq. len=1,0242026.05 | 7.653 | 1.7 | |
| MambaSeq. len=32,7682026.05 | 7.69 | — | |
| MambaSeq. len=1,0242026.05 | 7.722 | — | |
| HarmonicSeq. len=2,0482026.05 | 7.761 | 3.1 | |
| TransformerSeq. len=1,0242026.05 | 7.787 | — | |
| HarmonicSeq. len=4,0962026.05 | 7.792 | 4.2 | |
| HarmonicSeq. len=16,3842026.05 | 7.85 | 5.6 | |
| MambaSeq. len=2,0482026.05 | 7.868 | — | |
| HarmonicSeq. len=8,1922026.05 | 7.888 | 4.9 | |
| MambaSeq. len=4,0962026.05 | 7.895 | — | |
| MambaSeq. len=16,3842026.05 | 7.963 | — | |
| MambaSeq. len=8,1922026.05 | 7.988 | — | |
| TransformerSeq. len=2,0482026.05 | 8.012 | — | |
| TransformerSeq. len=32,7682026.05 | 8.114 | — | |
| TransformerSeq. len=4,0962026.05 | 8.136 | — | |
| TransformerSeq. len=8,1922026.05 | 8.292 | — | |
| TransformerSeq. len=16,3842026.05 | 8.319 | — | |
| Uniform Refine LSTM (UR)gate=refine, initialization=uniform2019.10 | 33.6 | — | |
| Uniform LSTM (U-)initialization=uniform2019.10 | 33.8 | — | |
| Ordered Master LSTM (OM)gate=master, initialization=ordered, activation=cumax2019.10 | 34 | — | |
| LSTM baseline2019.10 | 34.3 | — | |
| Chrono LSTMinitialization=chrono, T_max=82019.10 | 34.3 | — | |
| Chrono LSTMinitialization=chrono, T_max=112019.10 | 34.6 | — | |
| Chrono LSTMinitialization=chrono2019.10 | 35 | — |