Machine Translation on WMT 2014 (test)
45.6BLEUEdunov et al.
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Edunov et al.Supervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 45.6 | — | |
| NMT (transformer)Paradigm=Supervised2018.09 | 41.8 | — | |
| Vaswani et al.Supervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 41 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 36.2 | — | |
| WMT bestParadigm=Supervised2018.09 | 35.8 | — | |
| WMT bestSupervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 35.8 | — | |
| WMT bestParadigm=Supervised2018.09 | 35 | — | |
| WMT bestSupervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 35 | — | |
| Edunov et al.Supervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 35 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Detokenized (SacreBLEU)2019.02 | 33.6 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 33.5 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Detokenized (SacreBLEU)2019.02 | 33.2 | — | |
| Transformer-baseType=Autoregressive, Implementation=Ours2019.09 | 31.44 | — | |
| BiBERT2022.06 | 31.26 | — | |
| CMLM-baseKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 30.86 | — | |
| CMLMIteration=102021.03 | 30.86 | — | |
| SMT (europarl)Paradigm=Supervised2018.09 | 30.82 | — | |
| Bi-SimCut Pretrain + SimCut FinetunePre-train=Bi-SimCut, Fine-tuning=SimCut2022.06 | 30.78 | — | |
| CNAT (N=9)Iteration=1, candidates_reranked=92021.03 | 30.75 | 5.59 | |
| BERT-Fuse2022.06 | 30.75 | — | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=NPD, n=302019.09 | 30.68 | — | |
| SMT (europarl)Paradigm=Supervised, Lexical Reordering=Included, Vocabulary=Full, Tuning=Standard2018.09 | 30.61 | — | |
| BERT Initialization (12 layers)Model layers=12 layers2022.06 | 30.6 | — | |
| SimCut2022.06 | 30.56 | — | |
| SMT (europarl) + w/o lexical reord.Paradigm=Supervised, Lexical Reordering=Excluded2018.09 | 30.54 | — | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=NPD, n=152019.09 | 30.48 | — | |
| SMT (europarl) + w/o lexical reord.Paradigm=Supervised, Lexical Reordering=Excluded2018.09 | 30.33 | — | |
| R-Drop2022.06 | 30.13 | — | |
| CMLMIteration=42021.03 | 30.11 | — | |
| Bi-SimCut PretrainPre-train=Bi-SimCut2022.06 | 30.1 | — | |
| SMT (europarl) + constrained vocab.Paradigm=Supervised, Lexical Reordering=Excluded, Vocabulary=Constrained2018.09 | 30.1 | — | |
| SMT (europarl) + constrained vocab.Paradigm=Supervised, Lexical Reordering=Excluded, Vocabulary=Constrained2018.09 | 30.04 | — | |
| Evolved Transformer2022.06 | 29.8 | — | |
| CMLM-smallKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 29.47 | — | |
| SMT (europarl) + unsup. tuningParadigm=Supervised, Lexical Reordering=Excluded, Vocabulary=Constrained, Tuning=Unsupervised2018.09 | 29.46 | — | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=IWD, n=152019.09 | 29.44 | — | |
| CNATIteration=12021.03 | 29.36 | 10.37 | |
| B2T connectionEnc-Dec Layers=100L-100L2022.06 | 29.33 | — | |
| SMT (europarl) + unsup. tuningParadigm=Supervised, Lexical Reordering=Excluded, Vocabulary=Constrained, Tuning=Unsupervised2018.09 | 29.32 | — | |
| Transformer + Large BatchBatch size=Large2022.06 | 29.3 | — | |
| WMT bestSupervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 29 | — | |
| NAT-REGKnowledge Distillation=true, Decoding method=NPD, n=92019.09 | 28.9 | — | |
| Pre-LNEnc-Dec Layers=100L-100L2022.06 | 28.42 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=NPD, n=302019.09 | 28.4 | — | |
| Vaswani et al.Supervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 28.4 | — | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=NPD, n=302019.09 | 28.29 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=NPD, n=152019.09 | 28.07 | — | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=NPD, n=152019.09 | 27.71 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=IWD, n=152019.09 | 27.4 | — | |
| Transformer-baseType=Autoregressive2019.09 | 27.3 | — | |
| Transformer-baseType=Autoregressive, Implementation=Ours2019.09 | 27.16 | — | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=IWD, n=152019.09 | 27.16 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 27 | — | |
| CMLM-baseKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 26.92 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Detokenized (SacreBLEU)2019.02 | 26.4 | — | |
| Proposed systemParadigm=Unsupervised2018.09 | 26.22 | — | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=NPD, n=302019.09 | 26.04 | — | |
| Proposed systemParadigm=Unsupervised2018.09 | 25.87 | — | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=NPD, n=152019.09 | 25.76 | — | |
| CMLM-smallKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 25.51 | — | |
| NAT-IRKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 25.48 | — | |
| IR-NATIteration=102021.03 | 25.48 | 2.01 | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=NPD, n=302019.09 | 25.31 | — | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=NPD, n=152019.09 | 25.03 | — | |
| FlowSeqModel size=large, Knowledge Distillation=true, Decoding method=IWD, n=152019.09 | 24.7 | — | |
| CMLM-baseKnowledge Distillation=false, Decoding method=refinement, n=102019.09 | 24.65 | — | |
| NeoDiffModel type=Hybrid, Beam size=102025.05 | 24.64 | — | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=IWD, n=152019.09 | 24.63 | — | |
| NAT-REGKnowledge Distillation=true, Decoding method=NPD, n=92019.09 | 24.61 | — | |
| DiNoiSerModel type=Continuous, Beam size=502025.05 | 24.26 | — | |
| DiNoiSerModel type=Continuous, Beam size=52025.05 | 24.25 | — | |
| LV NARKnowledge Distillation=true, Decoding method=refinement, n=42019.09 | 24.2 | — | |
| NeoDiffModel type=Hybrid, Beam size=12025.05 | 24.09 | — | |
| IR-NATIteration=52021.03 | 23.86 | 3.11 | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=NPD, n=302019.09 | 23.64 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=NPD, n=302019.09 | 23.48 | — | |
| DifformerModel type=Continuous, Beam size=102025.05 | 23.26 | — | |
| CMLMModel type=Discrete, Beam size=52025.05 | 23.22 | — | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=NPD, n=152019.09 | 23.14 | — | |
| CMLM(MBR)Model type=Discrete, Beam size=52025.05 | 23.09 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=NPD, n=152019.09 | 23.08 | — | |
| FlowSeqModel size=large, Knowledge Distillation=false, Decoding method=IWD, n=152019.09 | 22.94 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 22.5 | — | |
| FlowSeqModel size=base, Knowledge Distillation=true, Decoding method=IWD, n=152019.09 | 22.49 | — | |
| NAT w/ FTKnowledge Distillation=true, Decoding method=NPD, n=102019.09 | 22.42 | — | |
| DifformerModel type=Continuous, Beam size=12025.05 | 22.13 | — | |
| CMLM-baseKnowledge Distillation=false, Decoding method=refinement, n=42019.09 | 22.06 | — | |
| NAT-IRKnowledge Distillation=true, Decoding method=refinement, n=102019.09 | 21.61 | — | |
| Proposed system (NMT hybridization)Supervision=Unsupervised, Evaluation Protocol=Detokenized (SacreBLEU)2019.02 | 21.2 | — | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=NPD, n=302019.09 | 21.15 | — | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=NPD, n=152019.09 | 20.81 | — | |
| WMT bestSupervision=Supervised, Evaluation Protocol=Tokenized (multi-bleu.perl)2019.02 | 20.6 | — | |
| IR-NATIteration=22021.03 | 20.39 | 8.77 | |
| FlowSeqModel size=base, Knowledge Distillation=false, Decoding method=IWD, n=152019.09 | 20.2 | — | |
| SeqDiffuSeqModel type=Continuous, Beam size=102025.05 | 19.76 | — | |
| SeqDiffuSeqModel type=Continuous, Beam size=12025.05 | 19.16 | — | |
| NAT w/ FTKnowledge Distillation=true, Decoding method=NPD, n=102019.09 | 18.66 | — | |
| DiffusionLMModel type=Continuous, Beam size=502025.05 | 17.41 | — | |
| IR-NATIteration=12021.03 | 16.77 | 11.39 | |
| DiffusionLMModel type=Continuous, Beam size=52025.05 | 15.33 | — |