Character-level Language Modeling on enwik8 (train)
0.72BPCRWKV-RNN
Evaluation Results
| Method | Links | |
|---|---|---|
| RWKV-RNNL=6, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Regularization=No weight decay nor dropout2023.05 | 0.72 | |
| TransformerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.05 | 0.977 | |
| Linear TransformerL=12, d=512, T=1024, Time Complexity=O(Td^2), Space Complexity=O(Td + d^2)2023.05 | 0.981 | |
| SynthesizerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.05 | 0.994 | |
| PerformerL=12, d=512, T=1024, Time Complexity=O(Td^2 log d), Space Complexity=O(Td log d + d^2 log d)2023.05 | 1.002 | |
| RWKV-RNNL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Optimizer=AdamW, Weight Decay=0.1, Dropout=0.1, Batch Size=16, Initial Learning Rate=6e-42023.05 | 1.01 | |
| TransformerL=24, d=256, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.05 | 1.039 | |
| ReformerL=12, d=512, T=1024, Time Complexity=O(T log T d), Space Complexity=O(T log T + Td)2023.05 | 1.04 | |
| AFT-simpleL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(Td)2023.05 | 1.046 | |
| Late-phase Rank1Number of units=500, SWA=false2020.07 | 1.522 | |
| Base + Rank1Number of units=500, SWA=false2020.07 | 1.524 | |
| Base LSTMNumber of units=500, SWA=false2020.07 | 1.57 |