Machine Translation on WMT En-Fr 2014 (test)
67.88BLEUN-shot
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| N-shotModel Size=13B2024.01 | 67.88 | — | — | — | — | — | — | |
| Batch-ICLModel Size=13B2024.01 | 67.63 | — | — | — | — | — | — | |
| N-shotModel Size=7B2024.01 | 66.38 | — | — | — | — | — | — | |
| Batch-ICLModel Size=7B2024.01 | 65.91 | — | — | — | — | — | — | |
| mRASPTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 45.4 | — | — | — | — | — | — | |
| mRASPTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 44.3 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=250L-250L2023.05 | 44.3 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=500L-500L2023.05 | 44.27 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=100L-100L2023.05 | 44.2 | — | — | — | — | — | — | |
| RealFormer#Encoder layers=60, #Decoder layers=122020.12 | 43.97 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=100L-100L2023.05 | 43.93 | — | — | — | — | — | — | |
| MixNormLN=Post, Depth=100L-100L2023.05 | 43.91 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=65536, heads=128, d_k=128, d_v=128, Parameters=2.89B, d_model=10242018.11 | 43.9 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=50L-50L2023.05 | 43.89 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=250L-250L2023.05 | 43.87 | — | — | — | — | — | — | |
| MixNormLN=Post, Depth=50L-50L2023.05 | 43.81 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=32768, heads=64, d_k=128, d_v=128, Parameters=1.48B, d_model=10242018.11 | 43.8 | — | — | — | — | — | — | |
| ADMIN#Encoder layers=60, #Decoder layers=122020.12 | 43.8 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=50L-50L2023.05 | 43.79 | — | — | — | — | — | — | |
| BERT-fused model2020.02 | 43.78 | — | — | — | — | — | — | |
| MixNormLN=Post, Depth=250L-250L2023.05 | 43.73 | — | — | — | — | — | — | |
| ADMIN (re-run)#Encoder layers=60, #Decoder layers=122020.12 | 43.72 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=500L-500L2023.05 | 43.67 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=18L-18L2023.05 | 43.53 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=16384, heads=32, d_k=128, d_v=128, Parameters=0.77B, d_model=10242018.11 | 43.5 | — | — | — | — | — | — | |
| mRASP2Training Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 43.5 | — | — | — | — | — | — | |
| Sub-LNLN=Pre, Depth=250L-250L2023.05 | 43.42 | — | — | — | — | — | — | |
| MixNormLN=Post, Depth=500L-500L2023.05 | 43.41 | — | — | — | — | — | — | |
| MixNormLN=Post, Depth=18L-18L2023.05 | 43.34 | — | — | — | — | — | — | |
| Sub-LNLN=Pre, Depth=100L-100L2023.05 | 43.31 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=8192, heads=16, d_k=128, d_v=128, Parameters=0.42B, d_model=10242018.11 | 43.3 | — | — | — | — | — | — | |
| Local Joint Self-attention2019.05 | 43.3 | — | — | — | — | — | — | |
| Sub-LNLN=Pre, Depth=50L-50L2023.05 | 43.28 | — | — | — | — | — | — | |
| Depth GrowingNumber of Blocks=82019.07 | 43.27 | — | — | — | — | — | — | |
| Vanilla Post-LNLN=Pre, Depth=18L-18L2023.05 | 43.27 | — | — | — | — | — | — | |
| Vanilla Pre-LNLN=Pre, Depth=250L-250L2023.05 | 43.25 | — | — | — | — | — | — | |
| Sub-LNLN=Pre, Depth=500L-500L2023.05 | 43.21 | — | — | — | — | — | — | |
| Ott et al. (2018)2019.05 | 43.2 | — | — | — | — | — | — | |
| Wu et al. (2019)2019.05 | 43.2 | — | — | — | — | — | — | |
| Joint Self-attention2019.05 | 43.2 | — | — | — | — | — | — | |
| Ott et al. (2018)2018.09 | 43.2 | — | — | — | — | — | — | |
| DynamicConv2020.02 | 43.2 | — | — | — | — | — | — | |
| Transformer-6Training Strategy=bilingual, Model Architecture (Layers)=6 layers2021.05 | 43.2 | — | — | — | — | — | — | |
| Vanilla Pre-LNLN=Pre, Depth=500L-500L2023.05 | 43.18 | — | — | — | — | — | — | |
| Vanilla Pre-LNLN=Pre, Depth=100L-100L2023.05 | 43.12 | — | — | — | — | — | — | |
| mRASPTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers, Fine-tuning Status=without fine-tune2021.05 | 43.1 | — | — | — | — | — | — | |
| Pre-LN#Encoder layers=60, #Decoder layers=122020.12 | 43.1 | — | — | — | — | — | — | |
| DLCLLN=Pre, Depth=50L-50L2023.05 | 43.05 | — | — | — | — | — | — | |
| Transformerbatch size=Large2020.02 | 43 | — | — | — | — | — | — | |
| Transformer-bigRole=Teacher2023.05 | 42.98 | — | — | 69.58 | — | — | — | |
| Transformerreproduced=true2020.02 | 42.96 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=18L-18L2023.05 | 42.92 | — | — | — | — | — | — | |
| DLCLLN=Pre, Depth=18L-18L2023.05 | 42.81 | — | — | — | — | — | — | |
| TransformerNumber of Blocks=102019.07 | 42.73 | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Large, Position Encoder Location=All blocks2020.03 | 42.7 | — | — | — | — | — | — | |
| Vanilla Pre-LNLN=Pre, Depth=50L-50L2023.05 | 42.7 | — | — | — | — | — | — | |
| TransformerNumber of Blocks=6, Source=Reproduced2019.07 | 42.69 | — | — | — | — | — | — | |
| Sub-LNLN=Pre, Depth=18L-18L2023.05 | 42.68 | — | — | — | — | — | — | |
| TransformerNumber of Blocks=82019.07 | 42.63 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=4096, heads=8, d_k=128, d_v=128, Parameters=0.24B, d_model=10242018.11 | 42.5 | — | — | — | — | — | — | |
| Vanilla Pre-LNLN=Pre, Depth=18L-18L2023.05 | 42.48 | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Large, Position Encoder Location=Input block only2020.03 | 42.4 | — | — | — | — | — | — | |
| Fixed-length Position EmbeddingModel Variant=Transformer-Large, Position Encoder Location=Input block only2020.03 | 42.4 | — | — | — | — | — | — | |
| Pretrained Transformer LargeInitialization=N/A2021.03 | 42.2 | — | — | — | — | — | — | |
| Transformer-MoSConfiguration=Big2018.09 | 42.1 | — | — | — | — | — | — | |
| T2RInitialization=Pretrain, Feature Size k (cross)=32, Feature Size k (causal)=4, Train Time (GPU hours)=82h2021.03 | 42.1 | — | — | — | — | — | — | |
| Pre-defined Sinusoidal Position EncoderModel Variant=Transformer-Large, Position Encoder Location=All blocks2020.03 | 42 | — | — | — | — | — | — | |
| Fixed-length Position EmbeddingModel Variant=Transformer-Large, Position Encoder Location=All blocks2020.03 | 42 | — | — | — | — | — | — | |
| m-TransformerTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 42 | — | — | — | — | — | — | |
| HS-NASLatency Constraint=200ms, LLM Predictor=GPT-4, Supernet=HAT, Iteration Range=1, 152023.10 | 42 | 0.1878 | — | — | 3.7 | 79.5 | 3.88 | |
| MixNormLN=Post, Depth=6L-6L2023.05 | 41.96 | — | — | — | — | — | — | |
| RealFormer#Encoder layers=6, #Decoder layers=62020.12 | 41.92 | — | — | — | — | — | — | |
| Mesh-TensorFlow Transformerd_ff=2048, heads=4, d_k=128, d_v=128, Parameters=0.15B, d_model=10242018.11 | 41.8 | — | — | — | — | — | — | |
| Transformerd_ff=4096, heads=16, d_k=64, d_v=64, Parameters=0.21B, d_model=10242018.11 | 41.8 | — | — | — | — | — | — | |
| Vaswani et al. (2017)2018.09 | 41.8 | — | — | — | — | — | — | |
| TransformerNumber of Blocks=6, Source=Reported2019.07 | 41.8 | — | — | — | — | — | — | |
| Pre-defined Sinusoidal Position EncoderModel Variant=Transformer-Large, Position Encoder Location=Input block only2020.03 | 41.8 | — | — | — | — | — | — | |
| HATHardware-Aware=true, Hetero. Layers=true, Latency=9.1s, #Params=57M, FLOPS (G)=3.9, GPU Hours=224, CO2e (lbs)=64, Cloud Comp. Cost=$166 - $5552020.05 | 41.8 | — | — | — | — | — | — | |
| ELUInitialization=Pretrain, Feature Size k (cross)=64, Feature Size k (causal)=64, Train Time (GPU hours)=80h2021.03 | 41.8 | — | — | — | — | — | — | |
| RFAInitialization=Pretrain, Feature Size k (cross)=32, Feature Size k (causal)=4, Train Time (GPU hours)=90h2021.03 | 41.8 | — | — | — | — | — | — | |
| Transformer (Vaswani et al.)Initialization=N/A2021.03 | 41.8 | — | — | — | — | — | — | |
| TransformerConfiguration=Big2018.09 | 41.7 | — | — | — | — | — | — | |
| NMT + MSOobjective=Margin-based Sentence-level Objective2021.05 | 41.7 | — | — | — | — | — | — | |
| RFAInitialization=Random Init., Feature Size k (cross)=32, Feature Size k (causal)=4, Train Time (GPU hours)=135h2021.03 | 41.7 | — | — | — | — | — | — | |
| Cascaded EncoderModel Architecture=Cascaded2018.04 | 41.67 | — | — | — | — | — | — | |
| BranchNormLN=Post, Depth=6L-6L2023.05 | 41.67 | — | — | — | — | — | — | |
| Multi-Column EncoderModel Architecture=MultiCol2018.04 | 41.66 | — | — | — | — | — | — | |
| ADMIN (re-run)#Encoder layers=6, #Decoder layers=62020.12 | 41.65 | — | — | — | — | — | — | |
| Gehring et al. (2017)2018.09 | 41.62 | — | — | — | — | — | — | |
| FLOATERModel Variant=Transformer-Base, Position Encoder Location=All blocks2020.03 | 41.6 | — | — | — | — | — | — | |
| Neuron-wise MoSLatency Constraint=200ms2023.10 | 41.6 | 0.1841 | — | — | 4.53 | 99.4 | 8.77 | |
| TIE-KDBase architecture=Transformer-base, Learning type=Knowledge Distillation2023.05 | 41.57 | — | — | 65.06 | — | — | — | |
| NMT + MTOobjective=Margin-based Token-level Objective2021.05 | 41.56 | — | — | — | — | — | — | |
| CBBGCABase architecture=Transformer-base, Learning type=Knowledge Distillation2023.05 | 41.54 | — | — | — | — | — | — | |
| Shaw et al. (2018)2019.05 | 41.5 | — | — | — | — | — | — | |
| Shaw, Uszkoreit, and Vaswani (2018)2018.09 | 41.5 | — | — | — | — | — | — | |
| HATLatency Constraint=200ms2023.10 | 41.5 | 0.1875 | — | — | 3.7 | 79.5 | 7.8 | |
| Vanilla Post-LNLN=Pre, Depth=6L-6L2023.05 | 41.48 | — | — | — | — | — | — | |
| DeepNormLN=Post, Depth=6L-6L2023.05 | 41.47 | — | — | — | — | — | — | |
| ADMIN#Encoder layers=6, #Decoder layers=62020.12 | 41.47 | — | — | — | — | — | — |