Machine Translation on IWSLT En-De 2014 (test)
37.85BLEUBaseline + relaxed cross attention
Evaluation Results
| Method | Links | |
|---|---|---|
| Baseline + relaxed cross attentionLanguage Model=MuST-C additional data, matched inference=false2022.09 | 37.85 | |
| Baseline + relaxed self-attentionLanguage Model=MuST-C additional data, matched inference=false2022.09 | 37.74 | |
| Baseline + relaxed self-attentionLanguage Model=MuST-C additional data, matched inference=true2022.09 | 37.71 | |
| Baseline + smoothed focusLanguage Model=MuST-C additional data2022.09 | 37.67 | |
| Baseline + relaxed self-attentionLanguage Model=None, matched inference=true2022.09 | 37.67 | |
| Baseline + relaxed self-attentionLanguage Model=IWSLT14 training transcripts, matched inference=true2022.09 | 37.67 | |
| Baseline + relaxed cross attentionLanguage Model=IWSLT14 training transcripts, matched inference=true2022.09 | 37.64 | |
| Baseline (Shen et al. resimulated)Language Model=MuST-C additional data2022.09 | 37.62 | |
| Shen et al.Language Model=None2022.09 | 37.6 | |
| Baseline + relaxed cross attentionLanguage Model=None, matched inference=true2022.09 | 37.6 | |
| Baseline + relaxed cross attentionLanguage Model=MuST-C additional data, matched inference=true2022.09 | 37.57 | |
| Baseline + relaxed self-attentionLanguage Model=None, matched inference=false2022.09 | 37.57 | |
| Baseline + relaxed cross attentionLanguage Model=None, matched inference=false2022.09 | 37.56 | |
| Baseline + relaxed cross attentionLanguage Model=IWSLT14 training transcripts, matched inference=false2022.09 | 37.53 | |
| Baseline + smoothed focusLanguage Model=IWSLT14 training transcripts2022.09 | 37.52 | |
| Baseline + relaxed self-attentionLanguage Model=IWSLT14 training transcripts, matched inference=false2022.09 | 37.49 | |
| Baseline (Shen et al. resimulated)Language Model=None2022.09 | 37.42 | |
| Baseline (Shen et al. resimulated)Language Model=IWSLT14 training transcripts2022.09 | 37.42 | |
| Baseline + smoothed focusLanguage Model=None2022.09 | 37.42 | |
| Liang et al.Language Model=None2022.09 | 37.25 | |
| Wu et al. [59]Language Model=None2022.09 | 36.88 | |
| AdminRemove LN=false, w_skip=true, Warmup=false, Optimizer=Adam2021.02 | 36.1 | |
| GradInitRemove LN=false, w_skip=true, Warmup=false, Optimizer=Adam2021.02 | 36.1 | |
| GradInitRemove LN=false, w_skip=false, Warmup=false, Optimizer=Adam2021.02 | 36 | |
| AdminRemove LN=false, w_skip=true, Warmup=true, Optimizer=RAdam2021.02 | 35.7 | |
| StandardRemove LN=false, w_skip=false, Warmup=true, Optimizer=RAdam2021.02 | 35.6 | |
| GradInitRemove LN=false, w_skip=true, Warmup=false, Optimizer=SGD2021.02 | 35.6 | |
| T-FixUpRemove LN=true, w_skip=true, Warmup=false, Optimizer=Adam2021.02 | 35.5 | |
| Wu et al. [58]Language Model=None2022.09 | 35.2 | |
| FixUpRemove LN=true, w_skip=false, Warmup=false, Optimizer=Adam2021.02 | 34.5 | |
| Fan et al.Language Model=None2022.09 | 34.5 | |
| Vaswani et al.Language Model=None2022.09 | 34.4 | |
| AdminRemove LN=false, w_skip=true, Warmup=false, Optimizer=SGD2021.02 | 33.7 | |
| Bi-SimCutbidirectional pretraining=true2022.06 | 31.16 | |
| Document-level + BERTContext level=document-level, BERT-fused=true2020.02 | 31.02 | |
| DRDAgranularity=dynamic2024.06 | 30.92 | |
| DRDAgranularity=static2024.06 | 30.84 | |
| R-Drop2022.06 | 30.73 | |
| CipherDAug - 2 keyssrc aug=false, tgt aug=false, |Θ|=3x2022.04 | 30.65 | |
| Data Diversesrc aug=true, tgt aug=true, |Θ|=7x2022.04 | 30.47 | |
| Sentence-level + BERTContext level=sentence-level, BERT-fused=true2020.02 | 30.45 | |
| BiBERTbidirectional pretraining=true2022.06 | 30.45 | |
| UniDropbase_model=Transformer2021.04 | 29.99 | |
| UniDrop2022.06 | 29.99 | |
| MixReps+co-teaching2021.04 | 29.93 | |
| Mixed Rep2022.06 | 29.93 | |
| MAT2021.04 | 29.9 | |
| Mixed-Repr.src aug=true, tgt aug=true, |Θ|=2x2022.04 | 29.71 | |
| VAT2022.06 | 29.45 | |
| NPRF enc-dec w/ RPEComplexity (encoder/decoder)=O(nlogn)/O(nlogn), Relative Positional Encoding=true2021.06 | 29.4 | |
| Standard enc + PRF decComplexity (encoder/decoder)=O(n²)/O(n), Encoder Attention=softmax, Decoder Attention=PRF2021.06 | 29.3 | |
| WordDropoutsrc aug=true, tgt aug=false, |Θ|=1x2022.04 | 29.21 | |
| Standard enc-decComplexity (encoder/decoder)=O(n²)/O(n²), Encoder Attention=softmax, Decoder Attention=softmax2021.06 | 29.2 | |
| CipherDAug - 1 keysrc aug=false, tgt aug=false, |Θ|=2x2022.04 | 29.14 | |
| RAML+SwitchOutsrc aug=true, tgt aug=true, |Θ|=1x2022.04 | 29.11 | |
| RAMLsrc aug=false, tgt aug=true, |Θ|=1x2022.04 | 29.07 | |
| Transformer2024.06 | 29.03 | |
| SwitchOutsrc aug=true, tgt aug=false, |Θ|=1x2022.04 | 29 | |
| Our Document-levelContext level=document-level, BERT-fused=false2020.02 | 28.9 | |
| Transformer-base+Self-ExplainingArchitecture=Transformer-base, Self-Explaining=true2020.12 | 28.9 | |
| Transformer w/ LABObeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 28.8 | |
| RAML+WordDropoutsrc aug=true, tgt aug=true, |Θ|=1x2022.04 | 28.78 | |
| Transformer2022.06 | 28.7 | |
| Transformer2021.04 | 28.67 | |
| BPE-Dropoutsrc aug=true, tgt aug=true, |Θ|=1x2022.04 | 28.63 | |
| Sentence-level baselineContext level=sentence-level, BERT-fused=false2020.02 | 28.57 | |
| Transformersrc aug=false, tgt aug=false, |Θ|=1x2022.04 | 28.57 | |
| Transformer-baseArchitecture=Transformer-base2020.12 | 28.4 | |
| Transformer w/ LSbeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 28.3 | |
| Transformer**Tokenization=BPE-based, Version=V1, Implementation=FairSeq, Number of parameters=48M2018.08 | 28.23 | |
| Transformer**Tokenization=BPE-based, Version=V2, Implementation=FairSeq, Number of parameters=52M2018.08 | 28.07 | |
| Pervasive AttentionTokenization=BPE-based, Version=V2, Number of parameters=22M2018.08 | 27.99 | |
| Miculicich et al. (2018)Context level=document-level, BERT-fused=false2020.02 | 27.94 | |
| Transformer w/ CPbeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 27.9 | |
| Transformer w/ AFLbeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 27.9 | |
| Transformerbeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 27.8 | |
| PRF enc-decComplexity (encoder/decoder)=O(n)/O(n), Encoder Attention=PRF, Decoder Attention=PRF2021.06 | 27.6 | |
| Pervasive AttentionTokenization=BPE-based, Version=V1, Number of parameters=11M2018.08 | 27.21 | |
| Transformer w/ FLbeam size=5, Backbone=6-layer encoder-decoder transformer2023.05 | 26.8 | |
| ConvS2S** (MLE)Tokenization=BPE-based, Version=V1, Implementation=FairSeq, Number of parameters=22M2018.08 | 26.73 | |
| CeMATInitialization=CeMAT, Architecture=Mask-Predict, Decoding Strategy=Non-autoregressive2022.03 | 26.7 | |
| NPMT + language modelTokenization=Word-based2018.08 | 25.36 | |
| RNNsearch*Tokenization=BPE-based, Version=V1, Implementation=our implementation, Number of parameters=15M2018.08 | 25.04 | |
| RIMsTraining Data=en -> de, en -> fr2019.09 | 24.23 | |
| TransformerDecoding Strategy=Autoregressive2022.03 | 23.9 | |
| mRASPInitialization=mRASP, Architecture=Mask-Predict, Decoding Strategy=Non-autoregressive2022.03 | 23.9 | |
| RIMsTraining Data=en -> de2019.09 | 23.71 | |
| TransformerTraining Data=en -> de, en -> fr2019.09 | 22.92 | |
| TransformerTraining Data=en -> de2019.09 | 22.89 | |
| Mask-PredictDecoding Strategy=Non-autoregressive2022.03 | 22 | |
| LSTMTraining Data=en -> de2019.09 | 21.32 | |
| LSTMTraining Data=en -> de, en -> fr2019.09 | 20.37 |