List your company
ResearchFormerly WizWand

Character-level Language Modeling on enwik8 (train)

0.72BPCRWKV-RNN
0.6860.91551.1451.3745Jul 25, 2020Jan 12, 2021Jul 3, 2021Dec 22, 2021Jun 12, 2022Dec 1, 2022May 22, 2023
Updated 8mo ago

Evaluation results

MethodLinks
RWKV-RNNL=6, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Regularization=No weight decay nor dropout2023.050.72
TransformerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.050.977
Linear TransformerL=12, d=512, T=1024, Time Complexity=O(Td^2), Space Complexity=O(Td + d^2)2023.050.981
SynthesizerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.050.994
PerformerL=12, d=512, T=1024, Time Complexity=O(Td^2 log d), Space Complexity=O(Td log d + d^2 log d)2023.051.002
RWKV-RNNL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Optimizer=AdamW, Weight Decay=0.1, Dropout=0.1, Batch Size=16, Initial Learning Rate=6e-42023.051.01
TransformerL=24, d=256, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.051.039
ReformerL=12, d=512, T=1024, Time Complexity=O(T log T d), Space Complexity=O(T log T + Td)2023.051.04
AFT-simpleL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(Td)2023.051.046
Late-phase Rank1Number of units=500, SWA=false2020.071.522
Base + Rank1Number of units=500, SWA=false2020.071.524
Base LSTMNumber of units=500, SWA=false2020.071.57