Word-level language modeling on WikiText-103 (dev)
15.72Perplexityadaptive kNN-LM
Evaluation results
| Method | Links | |
|---|---|---|
| adaptive kNN-LMn-gram overlap filtering=false2022.10 | 15.72 | |
| Adaptive kNN-LMλ=lambda_q, b=32, k=10242022.10 | 15.72 | |
| Adaptive kNN-LM (TFIDF)λ=lambda_q, b=32, k=10242022.10 | 15.76 | |
| kNN-LM + Continuous Cache# Trainable Params=247M2019.11 | 15.81 | |
| kNN-LM + CCacheλ=0.25, b=1, k=10242022.10 | 15.81 | |
| kNN-LM# Trainable Params=247M, base_model=Baevski & Auli (2019)2019.11 | 16.06 | |
| kNN-LMn-gram overlap filtering=false2022.10 | 16.06 | |
| kNN-LMλ=0.25, b=1, k=10242022.10 | 16.06 | |
| adaptive kNN-LMn-gram overlap filtering=true2022.10 | 17.26 | |
| kNN-LMn-gram overlap filtering=true2022.10 | 17.28 | |
| Continuous Cache# Trainable Params=247M2019.11 | 17.67 | |
| Transformer-XL Large#Params=257M2019.07 | 17.7 | |
| BASEnumber of layers=32, memory size (n)=null, evaluation context size=2562021.10 | 17.9 | |
| BASEn (memory size)=N/A, Number of layers=32, Evaluation context size=2562021.10 | 17.9 | |
| Baevski & Auli (2019)# Trainable Params=247M2019.11 | 17.96 | |
| Base LM (Baevski & Auli, 2019)# Trainable Params=247M2019.11 | 17.96 | |
| BaseLMn-gram overlap filtering=false2022.10 | 17.96 | |
| BaseLMn-gram overlap filtering=true2022.10 | 17.96 | |
| Base LMλ=-, b=-, k=-2022.10 | 17.96 | |
| BASEnumber of layers=16, memory size (n)=null, evaluation context size=4802021.10 | 18.4 | |
| ABCMLPnumber of layers=16, memory size (n)=64, evaluation context size=4802021.10 | 18.9 | |
| ABCMLPnumber of layers=32, memory size (n)=32, evaluation context size=2562021.10 | 19.2 | |
| ABCMLPn (memory size)=32, Number of layers=32, Evaluation context size=2562021.10 | 19.2 | |
| DeBERTa-MTModel scale=base, Pre-training tasks=MLM and ARLM2020.06 | 19.5 | |
| All-attention network + adaptive span#Params=133M2019.07 | 19.7 | |
| ABCMLPnumber of layers=16, memory size (n)=32, evaluation context size=4802021.10 | 19.7 | |
| BASEnumber of layers=16, memory size (n)=null, evaluation context size=02021.10 | 19.8 | |
| T2Rnumber of layers=32, memory size (n)=32, evaluation context size=2562021.10 | 20.1 | |
| T2Rn (memory size)=32, Number of layers=32, Evaluation context size=2562021.10 | 20.1 | |
| ABCMLPnumber of layers=16, memory size (n)=64, evaluation context size=02021.10 | 20.4 | |
| RFAnumber of layers=32, memory size (n)=32, evaluation context size=2562021.10 | 20.4 | |
| RFAn (memory size)=32, Number of layers=32, Evaluation context size=2562021.10 | 20.4 | |
| DeBERTaModel scale=base2020.06 | 20.5 | |
| DeBERTa-APModel scale=base, Position Embedding=Absolute (input layer)2020.06 | 20.7 | |
| ABCMLPnumber of layers=16, memory size (n)=32, evaluation context size=02021.10 | 21.2 | |
| RoBERTaModel scale=base2020.06 | 21.6 | |
| Skip Cross-Head Transformer-XLContext=2000, #Params=122M2023.11 | 21.87 | |
| RFA-GATE-Gaussian-StatefulModel Size=Big (~242M parameters)2021.03 | 22 | |
| ELUnumber of layers=32, memory size (n)=128, evaluation context size=2562021.10 | 22 | |
| ELUn (memory size)=128, Number of layers=32, Evaluation context size=2562021.10 | 22 | |
| ABCRDnumber of layers=16, memory size (n)=64, evaluation context size=4802021.10 | 22.3 | |
| Skip Cross-Head Transformer-XLContext=640, #Params=122M2023.11 | 22.88 | |
| Transformer-XL Standard#Params=151M2019.07 | 23.1 | |
| Transformer-XLModel scale=base2020.06 | 23.1 | |
| RFA-GATE-GaussianModel Size=Big (~242M parameters)2021.03 | 23.2 | |
| ABCRDnumber of layers=16, memory size (n)=64, evaluation context size=02021.10 | 23.2 | |
| Transformer-N#Params=148M2023.11 | 24.1 | |
| BASEModel Size=Big (~242M parameters)2021.03 | 24.5 | |
| RFA-GATE-arccosModel Size=Big (~242M parameters)2021.03 | 24.8 | |
| RFA-GaussianModel Size=Big (~242M parameters)2021.03 | 25.8 | |
| RFA-arccosModel Size=Big (~242M parameters)2021.03 | 26.4 | |
| Linformernumber of layers=16, memory size (n)=64, evaluation context size=02021.10 | 26.5 | |
| Linformernumber of layers=16, memory size (n)=64, evaluation context size=4802021.10 | 27.1 | |
| phi_eluModel Size=Big (~242M parameters)2021.03 | 28.7 | |
| LSTM + Hebbian + cache + MbPA2023.11 | 29 | |
| RFA-GATE-Gaussian-StatefulModel Size=Small (~38M parameters)2021.03 | 29.4 | |
| LSTM + Hebbian + Cache2019.07 | 29.7 | |
| RFA-GATE-GaussianModel Size=Small (~38M parameters)2021.03 | 31.3 | |
| 4-layer QRNN#Params=151M2019.07 | 32 | |
| RFA-GATE-arccosModel Size=Small (~38M parameters)2021.03 | 32.8 | |
| BASEModel Size=Small (~38M parameters)2021.03 | 33 | |
| RFA-GaussianModel Size=Small (~38M parameters)2021.03 | 33.6 | |
| RFA-arccosModel Size=Small (~38M parameters)2021.03 | 36 | |
| phi_eluModel Size=Small (~38M parameters)2021.03 | 38.4 |