Machine Translation on WMT En-De 2014 (test)
31.69BLEUTransformer
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TransformerKnowledge Distillation=true2021.03 | 31.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MATNd/d/dh=2/512/12288, #Param (M)=314.1, rho=0.22020.06 | 30.8 | — | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT-fused model2020.02 | 30.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mRASPTraining Size=4.5M, Pre-trained=true2022.03 | 30.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer (Ott et al., 2018) + LayerDropEnc Layers=12, Dec Layers=62019.09 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MATNd/d/dh=2/512/12288, #Param (M)=314.1, rho=0.32020.06 | 30.1 | — | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MATNd/d/dh=2/512/12288, #Param (M)=314.1, rho=02020.06 | 30 | — | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CeMATTraining Size=4.5M, Pre-trained=true2022.03 | 30 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Depth GrowingNumber of Blocks=82019.07 | 29.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MATNd/d/dh=2/512/12288, #Param (M)=314.1, rho=0.12020.06 | 29.9 | — | 29.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evolved TransformerParams=218M2019.01 | 29.8 | — | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evolved Transformer2020.02 | 29.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNd/d/dh=1/1024/4096, #Param (M)=325.7, rho=02020.06 | 29.8 | — | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| back-translationShallow Fusion=true, ILM Type=mini-self-attn2023.06 | 29.8 | — | — | — | — | — | — | — | — | 54.3 | — | — | — | — | — | |
| Wu et al. (2019)2019.05 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Joint Self-attention2019.05 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Local Joint Self-attention2019.05 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Wu et al.Params=213M2019.01 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DynamicConv (Wu et al., 2019)Enc Layers=7, Dec Layers=62019.09 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DynamicConv2020.02 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Wu et al.Param=213M2020.02 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-MoSConfiguration=Big2018.09 | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer (Ott et al., 2018) + LayerDropEnc Layers=6, Dec Layers=62019.09 | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TaLK ConvolutionParam=209M2020.02 | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer Big + Adversarial TrainingArchitecture=Transformer-Big2019.06 | 29.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evolved TransformerSize=DEEP, Parameters=218.1M2019.01 | 29.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNd/d/dh=1/512/12288, #Param (M)=288.9, rho=02020.06 | 29.5 | — | 29 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Feedback TransformerArchitecture Type=Transformer2020.02 | 29.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| back-translation2023.06 | 29.5 | — | — | — | — | — | — | — | — | 54.7 | — | — | — | — | — | |
| CNATKnowledge Distillation=true2021.03 | 29.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RealFormer#Encoder layers=18, #Decoder layers=182020.12 | 29.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ott et al. (2018)2019.05 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evolved TransformerSize=BIG, Parameters=221.7M2019.01 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ott et al.Params=210M2019.01 | 29.3 | — | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ott et al. (2018)2018.09 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer (Ott et al., 2018)Enc Layers=6, Dec Layers=62019.09 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformerbatch size=Large2020.02 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ott et al.Param=210M2020.02 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer (Ott et al., 2018)Architecture Type=Transformer2020.02 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DirectTraining Size=4.5M, Pre-trained=false2022.03 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Shaw et al. (2018)2019.05 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerSize=DEEP, Parameters=224.0M2019.01 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Shaw et al.Params=213M2019.01 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Shaw, Uszkoreit, and Vaswani (2018)2018.09 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Large, Position Encoder Location=All blocks2020.03 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Shaw et al.2020.02 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-BIG (En-De Teacher)Iteration=n/a, Speed=0.8x2021.06 | 29.2 | — | — | — | — | — | — | — | 73 | — | — | — | — | — | — | |
| TRANSFORMER-BIG + Localness + Rel_PosArchitecture=TRANSFORMER-BIG, Localness Modeling=True, Relative Position Encoding=True, # Para.=267.5M2018.10 | 29.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformerreproduced=true2020.02 | 29.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADMIN (re-run)#Encoder layers=18, #Decoder layers=182020.12 | 29.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerSize=BIG, Parameters=210.4M2019.01 | 29.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Large, Position Encoder Location=Input block only2020.03 | 29.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Kitaev et al.Param=213M2020.02 | 29.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baseline oursILM Type=mini-self-attn2023.06 | 29.1 | — | — | — | — | — | — | — | — | 54.8 | — | — | — | — | — | |
| RealFormer#Encoder layers=12, #Decoder layers=122020.12 | 29.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADMIN#Encoder layers=18, #Decoder layers=182020.12 | 29.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| He et al. (2018)2019.05 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER-BIG + D_RNN + O_CDRBackbone architecture=Transformer, Scale=Big, Components=DRNN + OCDR2018.11 | 28.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer Big + (Gao et al., 2019)Architecture=Transformer-Big2019.06 | 28.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNumber of Blocks=6, Source=Reproduced2019.07 | 28.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ahmed et al. (2017)2019.05 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ahmed et al.Params=213M2019.01 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Dehghani et al. (2018)2018.09 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ahmed et al.Param=213M2020.02 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BiARNArchitecture Type=Hybrid2020.02 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Model-level DL bigBidirectional=false, Parameters=210M x 22021.05 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pretrained Transformer LargeInitialization=N/A2021.03 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MASSTraining Size=4.5M, Pre-trained=true2022.03 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER-BIG + LocalnessArchitecture=TRANSFORMER-BIG, Localness Modeling=True, # Para.=267.4M2018.10 | 28.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADMIN (re-run)#Encoder layers=12, #Decoder layers=122020.12 | 28.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Multi-Column EncoderModel Architecture=MultiCol2018.04 | 28.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-bigRole=Teacher2023.05 | 28.81 | — | — | — | — | — | — | — | — | — | 53.2 | — | — | — | — | |
| baseline oursILM Type=separate LM2023.06 | 28.8 | — | — | — | — | — | — | — | — | 55.3 | — | — | — | — | — | |
| XLMTraining Size=4.5M, Pre-trained=true2022.03 | 28.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNumber of Blocks=82019.07 | 28.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerConfiguration=Big2018.09 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T2RInitialization=Pretrain, Feature Size k (cross)=32, Feature Size k (causal)=4, Train Time (GPU hours)=82h2021.03 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER-BIG + O_CDRBackbone architecture=Transformer, Scale=Big, Components=OCDR2018.11 | 28.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNumber of Blocks=102019.07 | 28.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cascaded EncoderModel Architecture=Cascaded2018.04 | 28.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Base, Position Encoder Location=All blocks2020.03 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MathNASHardware=Intel Xeon CPU, FLOPS=2.4G, Latency=272ms, Search Time=10s, CO2=0.0008lbs2023.11 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MathNASHardware=Raspberry Pi, FLOPS=2.8G, Latency=6.5s, Search Time=10s, CO2=0.0008lbs2023.11 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MathNASHardware=Nvidia TITAN Xp GPU, FLOPS=2.6G, Latency=189ms, Search Time=10s, CO2=0.0008lbs2023.11 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mBERTTraining Size=4.5M, Pre-trained=true2022.03 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER-BIG (baseline)Architecture=TRANSFORMER-BIG, # Para.=264.1M2018.10 | 28.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSOfull_name=Margin-based Sentence-level Objective, base=NMT + LM + MTO2021.05 | 28.58 | — | — | 1.36 | — | — | — | — | — | — | — | — | — | — | — | |
| ADMIN#Encoder layers=12, #Decoder layers=122020.12 | 28.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global-level Selectionimplementation=ours2021.05 | 28.57 | — | — | — | — | 1.28 | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER-BASE + Localness + Rel_PosArchitecture=TRANSFORMER-BASE, Localness Modeling=True, Relative Position Encoding=True, # Para.=88.9M2018.10 | 28.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chen et al. (2018)2019.05 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chen et al.Params=379M2019.01 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fixed-length Position EmbeddingModel Variant=Transformer-Large, Position Encoder Location=Input block only2020.03 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chen et al.Param=379M2020.02 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RNMT+Architecture Type=Recurrent2020.02 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HATHardware=Intel Xeon CPU, FLOPS=3.5G, Latency=384ms, Search Time=200h, CO2=57lbs2023.11 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MathNASHardware=Intel Xeon CPU, FLOPS=2.8G, Latency=336ms, Search Time=10s, CO2=0.0008lbs2023.11 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RNMT+Epochs=24.6, Training Time=40h2018.04 | 28.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RNMT+Model Architecture=RNMT+2018.04 | 28.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tay et al. (2020)2021.05 | 28.47 | — | — | — | — | 1.17 | — | — | — | — | — | — | — | — | — |