Language Modeling on Penn Treebank (PTB) (test)
14.72Perplexityfull-precision
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| full-precisionBits (W-E-A)=full-prec., Size (MB)=474.92022.03 | 14.72 | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 14.9 | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=4-4-8, Size (MB)=62.42022.03 | 14.95 | — | — | — | — | — | — | |
| LSQBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 15.43 | — | — | — | — | — | — | |
| LAQBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 15.87 | — | — | — | — | — | — | |
| PACTBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 16.11 | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=2-2-8, Size (MB)=33.02022.03 | 16.12 | — | — | — | — | — | — | |
| LAQBits (W-E-A)=4-4-8, Size (MB)=62.42022.03 | 16.55 | — | — | — | — | — | — | |
| LAQBits (W-E-A)=2-2-8, Size (MB)=33.02022.03 | 18.25 | — | — | — | — | — | — | |
| GPT2 + GFSA#Params=117M, Fine-tuning=true2023.12 | 19.45 | — | — | — | — | — | — | |
| GPT2#Params=117M, Fine-tuning=true2023.12 | 19.513 | — | — | — | — | — | — | |
| PACTBits (W-E-A)=4-4-8, Size (MB)=62.42022.03 | 20.17 | — | — | — | — | — | — | |
| HSNK (Symbols)=100, L (Walk steps)=52022.07 | 20.25 | — | — | — | — | — | 9.3 | |
| HSNK (Symbols)=50, L (Walk steps)=52022.07 | 22.47 | — | — | — | — | — | 9.5 | |
| FP16Bits=-, Backbone=LLAMA-2-7B2026.04 | 22.51 | — | — | — | — | — | — | |
| FP16Bits=–, Backbone=LLAMA-2-7B2026.04 | 22.51 | — | — | — | — | — | — | |
| Optimus Alambda=0.05, Optimized for=PPL2022.07 | 23.58 | — | — | — | — | — | 3.78 | |
| GPT2Fine-tuned=single epoch2022.07 | 24.23 | — | — | — | — | — | — | |
| HSNK (Symbols)=100, L (Walk steps)=202022.07 | 25.48 | — | — | — | — | — | 23.77 | |
| FBI-LLMBits=W1A16, Backbone=LLAMA-2-7B2026.04 | 29.6 | — | — | — | — | — | — | |
| HSNK (Symbols)=50, L (Walk steps)=202022.07 | 30.38 | — | — | — | — | — | 26.06 | |
| Optimus Blambda=1, Optimized for=MI2022.07 | 35.53 | — | — | — | — | — | 8.18 | |
| LBLLMBits=W(1+1)A4, Backbone=LLAMA-2-7B2026.04 | 36.79 | — | — | — | — | — | — | |
| LBLLMBits=W(1+1)A4, Backbone=LLAMA-2-7B2026.04 | 36.79 | — | — | — | — | — | — | |
| AWD-LSTM + MoS + Partial Shuffled + Adversarial TrainingParams=22M, dynamic evaluation=true2019.06 | 46.01 | — | — | — | — | — | — | |
| AWD-LSTM + MoS + Adversarial TrainingParams=22M, dynamic evaluation=true2019.06 | 46.52 | — | — | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluation with FRAGEParas=24M2018.09 | 46.54 | — | — | — | — | — | — | |
| AWD-LSTM + MoS + Partial ShuffledParams=22M, dynamic evaluation=true2019.06 | 47.49 | — | — | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, dynamic evaluation=true2019.06 | 47.69 | — | — | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluationParas=24M2018.09 | 47.69 | — | — | — | — | — | — | |
| AWD-LSTM + Adversarial TrainingParams=24M, dynamic evaluation=true2019.06 | 48.72 | — | — | — | — | — | — | |
| ADV-AWD-LSTM + MaxUpParams=24M, Dynamic evaluation=true2020.02 | 50.29 | — | — | — | — | — | — | |
| AWD-LSTMParams=24M, dynamic evaluation=true2019.06 | 51.1 | — | — | — | — | — | — | |
| ADV-AWD-LSTMParams=24M, Dynamic evaluation=true2020.02 | 51.1 | — | — | — | — | — | — | |
| AWD-LSTM + continuous cache pointer with FRAGEParas=24M2018.09 | 51.8 | — | — | — | — | — | — | |
| AWD-LSTM + MoS + Partial Shuffled + Adversarial TrainingParams=22M, dynamic evaluation=false2019.06 | 52.2 | — | — | — | — | — | — | |
| AWD-LSTM-DOC#Params=23M2022.03 | 52.4 | — | — | — | — | — | — | |
| AWD-LSTM + continuous cache pointerParas=24M2018.09 | 52.8 | — | — | — | — | — | — | |
| AWD-LSTM + MoS + Adversarial TrainingParams=22M, dynamic evaluation=false2019.06 | 52.87 | — | — | — | — | — | — | |
| AWD-LSTM-MoS with FRAGEParas=24M2018.09 | 53.31 | — | — | — | — | — | — | |
| iVAEMIEncoder/Decoder=one-layer LSTMs2022.07 | 53.44 | — | — | — | — | — | 12.5 | |
| AWD-LSTM + MoS + Partial ShuffledParams=22M, dynamic evaluation=false2019.06 | 53.92 | — | — | — | — | — | — | |
| ADV-AWD-LSTM + MaxUpParams=24M, Dynamic evaluation=false2020.02 | 54.27 | — | — | — | — | — | — | |
| LSTM + MoSSource=[49], Search Method=manual2019.02 | 54.4 | 22 | — | — | — | — | — | |
| AWD-LSTM-MoS#Params=22M2022.03 | 54.4 | — | — | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, dynamic evaluation=false2019.06 | 54.44 | — | — | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, Dynamic evaluation=false2020.02 | 54.44 | — | — | — | — | — | — | |
| AWD-LSTM-MoSParas=24M2018.09 | 54.44 | — | — | — | — | — | — | |
| AWD-FWM (ours)Number of parameters=24M, Seeds=32020.11 | 54.48 | — | — | — | — | — | — | |
| AWD-TXLNumber of parameters=24M, Seeds=3, AWD-style regularisation=true, Model averaging=true, Softmax temperature tuning=true2020.11 | 54.52 | — | — | — | — | — | — | |
| AWD-LSTM + Adversarial TrainingParams=24M, dynamic evaluation=false2019.06 | 55.01 | — | — | — | — | — | — | |
| ADV-AWD-LSTMParams=24M, Dynamic evaluation=false2020.02 | 55.01 | — | — | — | — | — | — | |
| AWD-LSTM-MoS w/o finetune with FRAGEParas=24M2018.09 | 55.23 | — | — | — | — | — | — | |
| Random search WSSource=Ours, Comparable Search Space=Y, Search Method=random2019.02 | 55.5 | 23 | 0.25 | 1 | 1.25 | — | — | |
| DARTS (second order)Source=[34], Comparable Search Space=Y, Search Method=gradient-based2019.02 | 55.7 | 23 | 1 | 1 | 2 | — | — | |
| DARTS (2nd)Params (M)=23, Search Cost (GPU days)=0.252019.10 | 55.7 | — | — | — | — | — | — | |
| DARTS (second order)Source=Ours (reproduced), Comparable Search Space=Y, Search Method=gradient-based2019.02 | 55.9 | 23 | 1 | 1 | 2 | — | — | |
| AWD-LSTM-MoS w/o finetuneParas=24M2018.09 | 55.97 | — | — | — | — | — | — | |
| LSTM + SEParams (M)=222019.10 | 56 | — | — | — | — | — | — | |
| DARTSParams=23M2019.06 | 56.1 | — | — | — | — | — | — | |
| AWD-LSTM + FRAGEParams=24M, Dynamic evaluation=false2020.02 | 56.1 | — | — | — | — | — | — | |
| AWD-LSTM with FRAGEParas=24M2018.09 | 56.1 | — | — | — | — | — | — | |
| ON-LSTM#Params=25M2022.03 | 56.2 | — | — | — | — | — | — | |
| DM-LSTM#Params=24M2022.03 | 56.2 | — | — | — | — | — | — | |
| ASHA + LSTM + DropConnectSource=[30], Comparable Search Space=N, Search Method=HP-tuned, Search Cost Unit=CPU-days2019.02 | 56.3 | 24 | — | — | 13 | — | — | |
| ENAS*Source=[41], Comparable Search Space=Y, Search Method=RL2019.02 | 56.3 | 24 | 0.5 | — | — | — | — | |
| ASHA baselineSource=Ours, Comparable Search Space=Y, Search Method=random2019.02 | 56.4 | 23 | — | — | 2 | — | — | |
| AWD-LSTMParams=24M, Dynamic evaluation=false2020.02 | 56.5 | — | — | — | — | — | — | |
| AWD-TXL (ours)Number of parameters=24M, Seeds=3, AWD-style regularisation=true, Model averaging=true, Softmax temperature tuning=true2020.11 | 56.5 | — | — | — | — | — | — | |
| Fraternal dropout#Params=24M2022.03 | 56.8 | — | — | — | — | — | — | |
| LSTM + DropConnectSource=[38], Search Method=manual2019.02 | 57.3 | 24 | — | — | — | — | — | |
| AWD-LSTMParams=24M, dynamic evaluation=false2019.06 | 57.3 | — | — | — | — | — | — | |
| AWD-LSTMParas=24M2018.09 | 57.3 | — | — | — | — | — | — | |
| AWD-LSTM#Params=24M2022.03 | 57.3 | — | — | — | — | — | — | |
| AWD-LSTM2022.05 | 57.3 | — | — | — | — | — | — | |
| AWD-LSTMNumber of parameters=24M, Seeds=32020.11 | 57.3 | — | — | — | — | — | — | |
| GDASParams (M)=23, Search Cost (GPU days)=0.42019.10 | 57.5 | — | — | — | — | — | — | |
| DARTS (first order)Source=[34], Comparable Search Space=Y, Search Method=gradient-based2019.02 | 57.6 | 23 | 0.5 | 1 | 1.5 | — | — | |
| DARTS (1st)Params (M)=23, Search Cost (GPU days)=0.132019.10 | 57.6 | — | — | — | — | — | — | |
| AWD-LSTM w/o finetune with FRAGEParas=24M2018.09 | 58 | — | — | — | — | — | — | |
| LSTM + SCParams (M)=242019.10 | 58.3 | — | — | — | — | — | — | |
| ENASSource=[34], Comparable Search Space=Y, Search Method=random2019.02 | 58.6 | 24 | 0.5 | — | — | — | — | |
| ENASParams (M)=24, Search Cost (GPU days)=0.52019.10 | 58.6 | — | — | — | — | — | — | |
| LSTMParams (M)=242019.10 | 58.8 | — | — | — | — | — | — | |
| AWD-LSTM w/o finetuneParas=24M2018.09 | 58.8 | — | — | — | — | — | — | |
| Random search baselineSource=[34], Comparable Search Space=Y, Search Method=random2019.02 | 59.4 | 23 | — | — | 2 | — | — | |
| PRPN2022.03 | 62 | — | — | — | — | — | — | |
| NAS-RNNParams=54M2019.06 | 62.4 | — | — | — | — | — | — | |
| NAS-RNNParams=54M, Dynamic evaluation=false2020.02 | 62.4 | — | — | — | — | — | — | |
| NASSource=[51], Comparable Search Space=N, Search Method=RL2019.02 | 64 | 25 | — | — | 10,000 | — | — | |
| NASParams (M)=25, Search Cost (GPU days)=10^42019.10 | 64 | — | — | — | — | — | — | |
| V-RHNParams (M)=232019.10 | 65.4 | — | — | — | — | — | — | |
| Variational RHN#Params=23M2022.03 | 65.4 | — | — | — | — | — | — | |
| Variational LSTM + weight tyingParams=51M2019.06 | 68.5 | — | — | — | — | — | — | |
| BWABits=W(1+1)A4, Backbone=LLAMA-2-7B2026.04 | 69.46 | — | — | — | — | — | — | |
| Pointer Sentinel-LSTM#Params=21M2022.03 | 70.9 | — | — | — | — | — | — | |
| Variational LSTMParams=19M2019.06 | 73.4 | — | — | — | — | — | — | |
| CBS-1-TModel Configuration=L22018.12 | 77.39 | — | — | — | — | 65 | — | |
| CBS-1Model Configuration=L22018.12 | 77.56 | — | — | — | — | 69 | — | |
| CBS-5Model Configuration=L22018.12 | 78.61 | — | — | — | — | 73 | — |