Language Modeling on WikiText-103 small setting (val)
31.8PerplexityDelta RNN
Evaluation results
| Method | Links | |
|---|---|---|
| Delta RNNSlow net=Feedforward, Update rule=delta, Fast net=RNN, Context truncation=Full context, #Prms=44.6M2021.06 | 31.8 | |
| RDNSlow net=Recurrent, Update rule=delta, Fast net=Linear, Context truncation=Full context, #Prms=44.1M2021.06 | 32.5 | |
| Delta LSTMSlow net=Feedforward, Update rule=delta, Fast net=LSTM, Context truncation=256 tokens, #Prms=47.3M2021.06 | 32.6 | |
| TransformerContext truncation=256 tokens, #Prms=44.0M2021.06 | 33 | |
| Delta RNNSlow net=Feedforward, Update rule=delta, Fast net=RNN, Context truncation=256 tokens, #Prms=44.6M2021.06 | 33.8 | |
| Delta Delta NetSlow net=Feedforward, Update rule=delta, Fast net=Delta Net, Context truncation=256 tokens, #Prms=44.6M2021.06 | 34 | |
| Delta NetSlow net=Feedforward, Update rule=delta, Fast net=Linear, Context truncation=256 tokens, #Prms=44.0M2021.06 | 34.1 | |
| RDNSlow net=Recurrent, Update rule=delta, Fast net=Linear, Context truncation=256 tokens, #Prms=44.1M2021.06 | 34.1 | |
| Delta MLPSlow net=Feedforward, Update rule=delta, Fast net=Deep MLP, Context truncation=256 tokens, #Prms=44.3M2021.06 | 35.8 | |
| Linear TransformerSlow net=Feedforward, Update rule=sum, Fast net=Linear, Context truncation=256 tokens, #Prms=44.0M2021.06 | 37.1 | |
| NAMEx-Full-MomParams=70M / 216M, Momentum=true2025.10 | 82.44 | |
| NAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 82.63 | |
| NAMEx-FullParams=70M / 216M, Momentum=false2025.10 | 82.85 | |
| EP-CAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 82.9 | |
| NAMExParams=70M / 216M, Momentum=false2025.10 | 83.3 | |
| CAMExParams=70M / 216M2025.10 | 83.53 | |
| EP-CAMExParams=70M / 216M, Momentum=false2025.10 | 83.89 | |
| SMoE (Top-2)Params=70M / 216M2025.10 | 84.26 | |
| SMEARParams=70M / 216M2025.10 | 85.56 | |
| SMoE (Top-1)Params=70M / 216M2025.10 | 86.64 |