Machine Translation on WMT En-De (newstest2014)
31.26BLEUBiBERT Contextualized Embeddings + Stochastic Layer Selection + Fine-Tuning
Evaluation Results
| Method | Links | |
|---|---|---|
| BiBERT Contextualized Embeddings + Stochastic Layer Selection + Fine-TuningStochastic Layer Selection=true, Fine-tuning Protocol=Fine-Tuning2021.09 | 31.26 | |
| BiBERT Contextualized Embeddings + Stochastic Layer SelectionStochastic Layer Selection=true2021.09 | 30.91 | |
| BERT-Fuse2021.09 | 30.75 | |
| BERT Initialization (12 layers)Model Architecture (Layers)=12 layers2021.09 | 30.6 | |
| BiBERT Contextualized Embeddings + Stochastic Layer Selection + Dual-Directional TrainingStochastic Layer Selection=true, Training Strategy=Dual-Directional Training2021.09 | 30.31 | |
| Evolved Transformer2021.09 | 29.8 | |
| Transformer + Large BatchTraining Strategy=Large Batch2021.09 | 29.3 | |
| Transformer+ALONE (Real number)D_inter=8192, Embed Params=8.4M2020.04 | 27.61 | |
| Transformer+ALONE (Binary)D_inter=8192, Embed Params=8.4M2020.04 | 27.55 | |
| TransformerEmbed Params=16.8M2020.04 | 27.3 | |
| Transformer (conventional word embeddings)Embed Params=16.8M2020.04 | 27.12 | |
| Transformer+DeFINE2020.04 | 27.01 | |
| Transformer+ALONE (Binary)D_inter=4096, Embed Params=4.2M2020.04 | 26.97 | |
| Transformer+ALONE (Real number)D_inter=8192, Embed Params=8.4M, Trainable Sigma=false2020.04 | 26.95 | |
| Transformer+ALONE (Real number)D_inter=4096, Embed Params=4.2M2020.04 | 26.93 | |
| Transformer+ALONE (Binary)D_inter=8192, Embed Params=8.4M, Trainable Sigma=false2020.04 | 26.9 | |
| Transformer+ALONE (Real number)D_inter=4096, Embed Params=4.2M, Trainable Sigma=false2020.04 | 26.85 | |
| Transformer+ALONE (Binary)D_inter=4096, Embed Params=4.2M, Trainable Sigma=false2020.04 | 26.75 | |
| Transformer (factorized embed)Embed Params=8.5M2020.04 | 26.56 | |
| Transformer (factorized embed)Embed Params=4.3M2020.04 | 26.43 | |
| RL-refined WPM-32Kensemble_size=8 models, vocabulary=32K wordpieces, RL_refinement=true2016.09 | 26.3 | |
| WPM-32Kensemble_size=8 models, vocabulary=32K wordpieces2016.09 | 26.2 | |
| ConvS2SEmbed Params=66.0M2020.04 | 25.2 | |
| GNMT (RNN Enc-Dec Att)Inputs=word-pieces, Outputs=word-pieces2016.10 | 24.61 | |
| GNMTSupervision=Supervised2017.10 | 24.61 | |
| ByteNetInputs=char, Outputs=char2016.10 | 23.75 | |
| GNMT (RNN Enc-Dec Att)Inputs=char, Outputs=char2016.10 | 22.62 | |
| RNN Enc-Dec Att (Chung et al.)Inputs=BPE, Outputs=char2016.10 | 21.33 | |
| RNN Enc-Dec Att (Luong et al.)Inputs=words, Outputs=words2016.10 | 20.9 | |
| Phrase Based MTInputs=phrases, Outputs=phrases2016.10 | 20.7 | |
| RNN Enc-Dec Att (Zhou et al.)Inputs=words, Outputs=words2016.10 | 20.6 | |
| RNN Enc-Dec Att (Chung et al.)Inputs=BPE, Outputs=BPE2016.10 | 19.98 | |
| Reverse RNN Enc-DecInputs=words, Outputs=words2016.10 | 14 | |
| RNN Enc-DecInputs=words, Outputs=words2016.10 | 11.3 | |
| Comparable NMT (full parallel)Supervision=Supervised, Parallel data=full2017.10 | 11.05 | |
| Proposed (full) + 100k parallelSupervision=Semi-supervised, Parallel data=100k2017.10 | 10.95 | |
| Proposed (full) + 10k parallelSupervision=Semi-supervised, Parallel data=10k2017.10 | 7.86 | |
| Proposed (+ BPE)Supervision=Unsupervised, Components=denoising, backtranslation, BPE2017.10 | 6.89 | |
| Proposed (+ backtranslation)Supervision=Unsupervised, Components=denoising, backtranslation2017.10 | 6.55 | |
| Comparable NMT (100k parallel)Supervision=Supervised, Parallel data=100k2017.10 | 5.29 | |
| Baseline (emb. nearest neighbor)Supervision=Unsupervised2017.10 | 4.39 | |
| Proposed (denoising)Supervision=Unsupervised, Components=denoising2017.10 | 2.4 | |
| Comparable NMT (10k parallel)Supervision=Supervised, Parallel data=10k2017.10 | 0.82 |