Language Modeling on WikiText-103 small setting (test)
32.8PerplexityDelta RNN
Evaluation results
| Method | Links | |
|---|---|---|
| Delta RNNSlow net=Feedforward, Update rule=delta, Fast net=RNN, Context truncation=Full context, #Prms=44.6M2021.06 | 32.8 | |
| RDNSlow net=Recurrent, Update rule=delta, Fast net=Linear, Context truncation=Full context, #Prms=44.1M2021.06 | 33.6 | |
| Delta LSTMSlow net=Feedforward, Update rule=delta, Fast net=LSTM, Context truncation=256 tokens, #Prms=47.3M2021.06 | 33.8 | |
| TransformerContext truncation=256 tokens, #Prms=44.0M2021.06 | 34.1 | |
| Delta RNNSlow net=Feedforward, Update rule=delta, Fast net=RNN, Context truncation=256 tokens, #Prms=44.6M2021.06 | 35 | |
| Delta NetSlow net=Feedforward, Update rule=delta, Fast net=Linear, Context truncation=256 tokens, #Prms=44.0M2021.06 | 35.2 | |
| Delta Delta NetSlow net=Feedforward, Update rule=delta, Fast net=Delta Net, Context truncation=256 tokens, #Prms=44.6M2021.06 | 35.2 | |
| RDNSlow net=Recurrent, Update rule=delta, Fast net=Linear, Context truncation=256 tokens, #Prms=44.1M2021.06 | 35.2 | |
| Delta MLPSlow net=Feedforward, Update rule=delta, Fast net=Deep MLP, Context truncation=256 tokens, #Prms=44.3M2021.06 | 36.8 | |
| Linear TransformerSlow net=Feedforward, Update rule=sum, Fast net=Linear, Context truncation=256 tokens, #Prms=44.0M2021.06 | 38.3 | |
| NAMEx-Full-MomParams=70M / 216M, Momentum=true2025.10 | 82.94 | |
| NAMEx-FullParams=70M / 216M, Momentum=false2025.10 | 83.16 | |
| NAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 83.59 | |
| EP-CAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 84.05 | |
| NAMExParams=70M / 216M, Momentum=false2025.10 | 84.12 | |
| CAMExParams=70M / 216M2025.10 | 84.48 | |
| SMoE (Top-2)Params=70M / 216M2025.10 | 84.81 | |
| EP-CAMExParams=70M / 216M, Momentum=false2025.10 | 85.03 | |
| SMEARParams=70M / 216M2025.10 | 87.24 | |
| SMoE (Top-1)Params=70M / 216M2025.10 | 87.79 |