Machine Translation on WMT En-De 2017 (test)
0.307BLEU ScoreBranchNorm
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BranchNormLN=Post, depth=50L-50L2023.05 | 0.307 | — | |
| BranchNormLN=Post, depth=18L-18L2023.05 | 0.303 | — | |
| BranchNormLN=Post, depth=100L-100L2023.05 | 0.298 | — | |
| DeepNormLN=Post, depth=50L-50L, implementation=reimplementation2023.05 | 0.297 | — | |
| BranchNormLN=Post, depth=250L-250L2023.05 | 0.296 | — | |
| EIT2022.12 | 0.2958 | 0.8183 | |
| DeepNormLN=Post, depth=100L-100L, implementation=reimplementation2023.05 | 0.293 | — | |
| BranchNormLN=Post, depth=6L-6L2023.05 | 0.293 | — | |
| SQ-Transformertraining_mode=SRL2024.02 | 0.2921 | — | |
| DeepNormLN=Post, depth=18L-18L, implementation=reimplementation2023.05 | 0.291 | — | |
| DeepNormLN=Post, depth=50L-50L2023.05 | 0.29 | — | |
| DeepNormLN=Post, depth=250L-250L, implementation=reimplementation2023.05 | 0.29 | — | |
| DeepNormLN=Post, depth=100L-100L2023.05 | 0.289 | — | |
| AdminLN=Post, depth=18L-18L2023.05 | 0.288 | — | |
| DeepNormLN=Post, depth=18L-18L2023.05 | 0.288 | — | |
| Sub-LNLN=Pre, depth=50L-50L, implementation=reimplementation2023.05 | 0.287 | — | |
| DeepNormLN=Post, depth=6L-6L, implementation=reimplementation2023.05 | 0.286 | — | |
| Transformer2022.12 | 0.2859 | 0.8111 | |
| R-FixupLN=No, depth=18L-18L2023.05 | 0.284 | — | |
| T-FixupLN=No, depth=18L-18L2023.05 | 0.284 | — | |
| NormFormerLN=Pre, depth=18L-18L2023.05 | 0.283 | — | |
| Sub-LNLN=Pre, depth=18L-18L, implementation=reimplementation2023.05 | 0.283 | — | |
| DLCLLN=Pre, depth=18L-18L2023.05 | 0.282 | — | |
| Vanilla Post-LNLN=Post, depth=6L-6L2023.05 | 0.281 | — | |
| Vanilla Pre-LNLN=Pre, depth=18L-18L2023.05 | 0.281 | — | |
| Transformer2024.02 | 0.281 | — | |
| Vanilla Pre-LNLN=Pre, depth=50L-50L2023.05 | 0.28 | — | |
| DS-InitLN=Post, depth=6L-6L2023.05 | 0.279 | — | |
| AdminLN=Post, depth=6L-6L2023.05 | 0.279 | — | |
| T-FixupLN=No, depth=50L-50L2023.05 | 0.279 | — | |
| Sub-LNLN=Pre, depth=250L-250L, implementation=reimplementation2023.05 | 0.279 | — | |
| NormFormerLN=Pre, depth=50L-50L2023.05 | 0.278 | — | |
| DeepNormLN=Post, depth=6L-6L2023.05 | 0.278 | — | |
| R-FixupLN=No, depth=50L-50L2023.05 | 0.277 | — | |
| DLCLLN=Pre, depth=250L-250L2023.05 | 0.277 | — | |
| Sub-LNLN=Pre, depth=100L-100L, implementation=reimplementation2023.05 | 0.277 | — | |
| R-FixupLN=No, depth=6L-6L2023.05 | 0.275 | — | |
| T-FixupLN=No, depth=6L-6L2023.05 | 0.275 | — | |
| Vanilla Pre-LNLN=Pre, depth=250L-250L2023.05 | 0.275 | — | |
| DLCLLN=Pre, depth=100L-100L2023.05 | 0.275 | — | |
| Sub-LNLN=Pre, depth=6L-6L, implementation=reimplementation2023.05 | 0.275 | — | |
| Vanilla Pre-LNLN=Pre, depth=100L-100L2023.05 | 0.274 | — | |
| DLCLLN=Pre, depth=6L-6L2023.05 | 0.274 | — | |
| Vanilla Pre-LNLN=Pre, depth=6L-6L2023.05 | 0.27 | — | |
| NormFormerLN=Pre, depth=6L-6L2023.05 | 0.27 | — | |
| ReZeroLN=No, depth=6L-6L2023.05 | 0.269 | — | |
| Tsenior → [Tjunior → S]Description=Student model distilled from senior teacher via EKD2026.05 | 0.1714 | 0.48 | |
| TjuniorDescription=Junior teacher model2026.05 | 0.169 | 0.47 | |
| Tjunior → SDescription=Student model distilled from junior teacher2026.05 | 0.1476 | 0.45 | |
| SDescription=Base student model2026.05 | 0.1146 | 0.42 |