Machine Translation on OPUS-100 (test)
39.1Average BLEU ScoreTrue (Ground Truth)
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| True (Ground Truth)Segmentation Protocol=Ground Truth, MT Model=OPUS-MT, Paragraph Size=10 sentences2023.05 | 39.1 | — | 44 | — | 48.4 | — | — | — | — | — | 37.5 | — | — | — | 48.8 | 37.7 | 43.7 | 28.9 | 39.2 | 18.6 | 22.3 | 54.1 | 29.1 | 27.1 | 61.2 | — | — | — | — | — | — | — | — | — | — | — | |
| WtPPUNCTSegmentation Protocol=WtPPUNCT model, MT Model=OPUS-MT, Paragraph Size=10 sentences2023.05 | 37.6 | — | 40.8 | — | 47 | — | — | — | — | — | 37.2 | — | — | — | 47.2 | 37.6 | 43.2 | 28.8 | 32.7 | 16.2 | 21.5 | 53.6 | 29 | 22.6 | 56.8 | — | — | — | — | — | — | — | — | — | — | — | |
| ErsatzSegmentation Protocol=Ersatz segmentation tool, MT Model=OPUS-MT, Paragraph Size=10 sentences2023.05 | 35.3 | — | 38 | — | 44.8 | — | — | — | — | — | 36.8 | — | — | — | 44.8 | 36.6 | 43.1 | 28.3 | 28.1 | 7.9 | — | 51.3 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BranchNorm#Layers=1000, #Params=3.8B2023.05 | 32.3 | — | — | — | — | — | — | — | — | — | — | — | — | 35 | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepNorm#Layers=1000, #Params=3.8B2023.05 | 32.1 | — | — | — | — | — | — | — | — | — | — | — | — | 33.9 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepNorm (reimplemented)#Layers=1000, #Params=3.8B2023.05 | 32.1 | — | — | — | — | — | — | — | — | — | — | — | — | 34.8 | 29.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BranchNorm#Layers=200, #Params=863M2023.05 | 31.4 | — | — | — | — | — | — | — | — | — | — | — | — | 34.2 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepNorm#Layers=200, #Params=863M2023.05 | 31.1 | — | — | — | — | — | — | — | — | — | — | — | — | 33.2 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepNorm (reimplemented)#Layers=200, #Params=863M2023.05 | 31.1 | — | — | — | — | — | — | — | — | — | — | — | — | 33.9 | 28.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pre-LN#Layers=1000, #Params=3.8B2023.05 | 31 | — | — | — | — | — | — | — | — | — | — | — | — | 34 | 28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pre-LN#Layers=200, #Params=863M2023.05 | 30.5 | — | — | — | — | — | — | — | — | — | — | — | — | 34.6 | 26.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline#Layers=48, #Params=254M2023.05 | 27.7 | — | — | — | — | — | — | — | — | — | — | — | — | 31.4 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NaïveSegmentation Protocol=10 equally long sequences, MT Model=OPUS-MT, Paragraph Size=10 sentences2023.05 | 26.7 | — | 29.4 | — | 35.6 | — | — | — | — | — | 31.5 | — | — | — | 26.7 | 25.5 | 33.5 | 20.1 | 21 | 9 | 11.6 | 43.4 | 18.7 | 21.4 | 31.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline#Layers=24, #Params=173M2023.05 | 26.2 | — | — | — | — | — | — | — | — | — | — | — | — | 29.5 | 22.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline#Layers=12, #Params=133M2023.05 | 24.5 | — | — | — | — | — | — | — | — | — | — | — | — | 27.5 | 21.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NoneSegmentation Protocol=Entire paragraph (no splitting), MT Model=OPUS-MT, Paragraph Size=10 sentences2023.05 | 22.8 | — | 16.4 | — | 27.4 | — | — | — | — | — | 31.8 | — | — | — | 21.8 | 30.6 | 37.3 | 20.8 | 8.7 | 3.2 | 0.6 | 30.3 | 22.4 | 4.4 | 17.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Pivotmode=pivot-based (X->En then En->Y), base_model=m-Transformer2021.05 | 15.56 | 5.5 | 17 | 28.5 | 16.4 | 2.2 | 6 | 26.1 | 22.3 | 14.4 | 14.2 | 16.6 | 19.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mRASP2mode=zero-shot, tokenization=de-tokenized (sacreBLEU)2021.05 | 15.31 | 5.3 | 17.3 | 29 | 14.5 | 5.3 | 6.1 | 23.6 | 21.7 | 12.3 | 15 | 16.4 | 19.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| m-Transformermode=zero-shot, tokenization=de-tokenized (sacreBLEU)2021.05 | 5.05 | 3.7 | 5.6 | 6.7 | 4.1 | 2.3 | 6.3 | 7.7 | 4.8 | 4.2 | 4.8 | 5.7 | 4.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DirectMode=Fine-tuning2020.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.5 | 0.6 | 0 | 0.2 | 6.1 | 5.8 | 11.7 | 10.1 | 24.2 | 26.2 | 4.5 | |
| mRASPMode=Fine-tuning2020.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 13.4 | 16.2 | 1.8 | 7.3 | 21.3 | 25.5 | 30.7 | 32.9 | 25.2 | 27.5 | 18.6 |