Language Modeling on Yahoo (test)
326.7NLLOurs + anneal
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Ours + annealObjective=Modified VAE Objective, Annealing=true2019.01 | 326.7 | 5.7 | — | 2.9 | 15 | |
| SA-VAE + annealObjective=Modified VAE Objective, Annealing=true2019.01 | 327.2 | 5.2 | — | 2.7 | 9.8 | |
| SA-VAEBackbone=LSTM, K=202018.02 | 327.5 | 7.19 | 60.4 | — | — | |
| SA-VAE + annealCategory=Previous Reports2019.01 | 327.5 | 7.19 | — | — | — | |
| SA-VAEBackbone=LSTM, K=102018.02 | 327.6 | 5.13 | 60.5 | — | — | |
| LSTM-LM*Objective=Standard VAE Objective2019.01 | 328 | — | — | — | — | |
| OursObjective=Standard VAE Objective2019.01 | 328.2 | 5.6 | — | 3 | 8 | |
| β-VAEObjective=Modified VAE Objective, beta=0.62019.01 | 328.5 | 0.3 | — | 0.2 | 1 | |
| VAE + annealObjective=Modified VAE Objective, Annealing=true2019.01 | 328.6 | 0 | — | 0 | 0 | |
| β-VAEObjective=Modified VAE Objective, beta=0.42019.01 | 328.7 | 6.3 | — | 2.8 | 8 | |
| β-VAEObjective=Modified VAE Objective, beta=0.82019.01 | 328.8 | 0 | — | 0 | 0 | |
| VAEObjective=Standard VAE Objective2019.01 | 329 | 0 | — | 0 | 0 | |
| LMBackbone=LSTM2018.02 | 329.1 | — | 61.6 | — | — | |
| SA-VAEObjective=Standard VAE Objective2019.01 | 329.2 | 0.1 | — | 0.1 | 0.8 | |
| SVIBackbone=LSTM, K=402018.02 | 329.8 | 1.01 | 62.2 | — | — | |
| VAE + SVI + KLBackbone=LSTM, K=20, Reference=Salakhutdinov & Larochelle (2010); Hjelm et al. (2016)2018.02 | 330.1 | 7.81 | 62.3 | — | — | |
| VAEBackbone=LSTM2018.02 | 330.2 | 0.01 | 62.5 | — | — | |
| VAE + SVI + KLBackbone=LSTM, K=10, Reference=Salakhutdinov & Larochelle (2010); Hjelm et al. (2016)2018.02 | 330.3 | 7.95 | 62.5 | — | — | |
| VAE + INITBackbone=LSTM, Initialization=Pretrained Language Model2018.02 | 330.5 | 0.37 | 62.7 | — | — | |
| VAE + SVIBackbone=LSTM, K=20, Reference=Krishnan et al. (2018)2018.02 | 330.5 | 7.8 | 62.7 | — | — | |
| SVIBackbone=LSTM, K=202018.02 | 330.8 | 0.41 | 62.9 | — | — | |
| VAE + SVIBackbone=LSTM, K=10, Reference=Krishnan et al. (2018)2018.02 | 331.2 | 7.85 | 63.3 | — | — | |
| SVIBackbone=LSTM, K=102018.02 | 331.4 | 0.16 | 63.4 | — | — | |
| LCNN-VAE + initEffective Filter Size=125, Pretraining=Only encoder using LSTM LM2017.02 | 332.1 | 10 | 63.9 | — | — | |
| CNN-VAE + INITBackbone=CNN, Initialization=Pretrained Language Model, Reference=Yang et al. (2017)2018.02 | 332.1 | 10 | 63.9 | — | — | |
| CNN-VAECategory=Previous Reports2019.01 | 332.1 | 10 | — | — | — | |
| β-VAEObjective=Modified VAE Objective, beta=0.22019.01 | 332.2 | 19.1 | — | 3.3 | 20.4 | |
| LCNN-VAEEffective Filter Size=1252017.02 | 333.9 | 6.7 | 65.4 | — | — | |
| CNN-VAEBackbone=CNN, Reference=Yang et al. (2017)2018.02 | 333.9 | 6.7 | 65.4 | — | — | |
| VAE + WORD-DROP 25%Backbone=LSTM, Word-dropout=25%2018.02 | 334.2 | 1.44 | 65.6 | — | — | |
| MCNN-VAE + initEffective Filter Size=63, Pretraining=Only encoder using LSTM LM2017.02 | 334.6 | 12.6 | 66 | — | — | |
| LSTM-LMEffective Filter Size=N/A2017.02 | 334.9 | — | 66.2 | — | — | |
| LSTM-LMBackbone=LSTM, Reference=Yang et al. (2017)2018.02 | 334.9 | — | 66.2 | — | — | |
| LCNN-LMEffective Filter Size=1252017.02 | 335.4 | — | 66.6 | — | — | |
| CNN-LMBackbone=CNN, Reference=Yang et al. (2017)2018.02 | 335.4 | — | 66.6 | — | — | |
| VLCNN-VAE + initEffective Filter Size=187, Pretraining=Only encoder using LSTM LM2017.02 | 335.8 | 3.8 | 67 | — | — | |
| SCNN-VAE + initEffective Filter Size=15, Pretraining=Only encoder using LSTM LM2017.02 | 335.9 | 13.9 | 67 | — | — | |
| MCNN-VAEEffective Filter Size=632017.02 | 336.2 | 11.8 | 67.3 | — | — | |
| VLCNN-LMEffective Filter Size=1872017.02 | 336.5 | — | 67.6 | — | — | |
| VLCNN-VAEEffective Filter Size=1872017.02 | 336.5 | 0.7 | 67.6 | — | — | |
| SCNN-VAEEffective Filter Size=152017.02 | 337.8 | 13.3 | 68.7 | — | — | |
| MCNN-LMEffective Filter Size=632017.02 | 338.3 | — | 69.1 | — | — | |
| LSTM-VAE** + initEffective Filter Size=N/A, Pretraining=Only encoder using LSTM LM2017.02 | 339.2 | 0 | 69.9 | — | — | |
| LSTM-VAE + INITBackbone=LSTM, Initialization=Pretrained Language Model, Reference=Yang et al. (2017)2018.02 | 339.2 | 0 | 69.9 | — | — | |
| LSTM-VAE**Effective Filter Size=N/A2017.02 | 342.1 | 0 | 72.5 | — | — | |
| LSTM-VAEBackbone=LSTM, Reference=Yang et al. (2017)2018.02 | 342.1 | 0 | 72.5 | — | — | |
| VAE + WORD-DROP 50%Backbone=LSTM, Word-dropout=50%2018.02 | 345 | 5.29 | 75.2 | — | — | |
| SCNN-LMEffective Filter Size=152017.02 | 345.3 | — | 75.5 | — | — | |
| GPT2Fine-tuned=single epoch2022.07 | — | — | 22 | — | — | |
| HSNK (Symbols)=50, L (Walk steps)=52022.07 | — | — | 20.99 | 10.42 | — | |
| HSNK (Symbols)=50, L (Walk steps)=202022.07 | — | — | 22.84 | 22.81 | — | |
| HSNK (Symbols)=100, L (Walk steps)=52022.07 | — | — | 21.01 | 11.21 | — | |
| HSNK (Symbols)=100, L (Walk steps)=202022.07 | — | — | 21.98 | 16.13 | — | |
| iVAEMIEncoder/Decoder=one-layer LSTMs2022.07 | — | — | 47.93 | 10.7 | — | |
| Optimus Alambda=0.05, Optimized for=PPL2022.07 | — | — | 22.34 | 5.34 | — | |
| Optimus Blambda=1, Optimized for=MI2022.07 | — | — | 29.92 | 9.18 | — |