Machine Translation on WMT16 EN-RO (test)
39.1BLEUMASS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MASSTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 39.1 | — | |
| mRASP2Training Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 39.1 | — | |
| mRASPTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 38.9 | — | |
| mBARTTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 38.8 | — | |
| mRASPTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers, Fine-tuning Status=without fine-tune2021.05 | 38.8 | — | |
| XLMTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 38.5 | — | |
| mRASP2Training Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 38 | — | |
| mBARTTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 37.7 | — | |
| m-TransformerTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 37.7 | — | |
| mRASPTraining Strategy=pre-train & fine-tuned, Fine-tuning Status=fine-tuned2021.05 | 37.6 | — | |
| mRASPTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers, Fine-tuning Status=without fine-tune2021.05 | 37.5 | — | |
| Transformer-12Training Strategy=bilingual, Model Architecture (Layers)=12 layers2021.05 | 36.8 | — | |
| m-TransformerTraining Strategy=unified multilingual, Model Architecture (Layers)=12 layers2021.05 | 35.9 | — | |
| Multi-DistillationTraining Strategy=unified multilingual2021.05 | 35.8 | — | |
| Transformer-6Training Strategy=bilingual, Model Architecture (Layers)=6 layers2021.05 | 34.3 | — | |
| Transformer-12Training Strategy=bilingual, Model Architecture (Layers)=12 layers2021.05 | 34.3 | — | |
| Transformer-6Training Strategy=bilingual, Model Architecture (Layers)=6 layers2021.05 | 34 | — | |
| Transformer-base (KD teacher)Bidirectional=false, Parameters=62M x 2, Speedup=1.0x2021.05 | 33.85 | — | |
| GLAT+CTCBidirectional=false, Parameters=62M x 2, Speedup=16.8x, Decoding=greedy2021.05 | 33.71 | — | |
| duplex REDER (final model)Bidirectional=true, Parameters=58M, Speedup=5.5x, Decoding=beam search (size 20)2021.05 | 33.6 | — | |
| Mask-PredictIdec=10, type=Iterative NAT2020.08 | 33.08 | 1.7 | |
| GLAT + NPDIdec=1, m=72020.08 | 32.87 | 7.9 | |
| GLATBidirectional=false, Parameters=62M x 2, Speedup=15.3x, Decoding=greedy2021.05 | 32.87 | — | |
| GLAT + CTCIdec=12020.08 | 32.79 | 14.6 | |
| MGNMTBidirectional=true, Parameters=195M2021.05 | 32.7 | — | |
| simplex REDERBidirectional=false, Parameters=58M x 2, Speedup=5.5x, Decoding=beam search (size 20)2021.05 | 32.67 | — | |
| CTC*Idec=12020.08 | 32.6 | 14.6 | |
| ImputerIdec=12020.08 | 32.3 | 18.6 | |
| ImputerBidirectional=false, Parameters=58M x 2, Speedup=18.6x, Decoding=greedy2021.05 | 32.3 | — | |
| Flowseq + NPDIdec=1, m=302020.08 | 32.2 | — | |
| CTCBidirectional=false, Parameters=58M x 2, Speedup=18.6x, Decoding=greedy2021.05 | 32.2 | — | |
| Multi-DistillationTraining Strategy=unified multilingual2021.05 | 31.6 | — | |
| imit-NAT + NPDIdec=1, m=72020.08 | 31.45 | 9.7 | |
| Transformerimplementation=ours2020.08 | 31.27 | 1 | |
| GLATIdec=12020.08 | 31.19 | 15.3 | |
| NAT-FT + NPDIdec=1, m=1002020.08 | 29.79 | 2.4 | |
| FlowseqIdec=12020.08 | 29.73 | 1.1 | |
| NAT-IRIdec=102020.08 | 29.32 | 1.5 | |
| imit-NATIdec=12020.08 | 28.61 | 18.6 | |
| NAT-base*Idec=12020.08 | 28.47 | 15.3 | |
| Mask-PredictIdec=1, type=Fully NAT2020.08 | 27.32 | — | |
| NAT-FTIdec=12020.08 | 27.29 | 15.6 | |
| vanilla NATBidirectional=false, Parameters=62M x 2, Speedup=15.6x, Decoding=greedy2021.05 | 27.29 | — | |
| mBART-largeBit-width=32-32-32, Layer-config=12-12, Compression-ratio=1x2022.03 | 26.82 | — | |
| DQ-mBARTProtocol=Distillation-Aware Quantization, Bit-width=8-8-8, Layer-config=12-12, Compression-ratio=4.0x2022.03 | 25.91 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=8-8-8, Layer-config=12-6, Compression-ratio=4.7x2022.03 | 25.61 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=8-8-8, Layer-config=12-3, Compression-ratio=5.2x2022.03 | 24.22 | — | |
| DQ-mBARTProtocol=Distillation-Aware Quantization, Bit-width=2-2-8, Layer-config=12-12, Compression-ratio=15.2x2022.03 | 23.48 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=8-8-8, Layer-config=12-1, Compression-ratio=5.6x2022.03 | 20.61 | — | |
| NAT-CTCIdec=12020.08 | 19.54 | — | |
| CTC w/o KDBidirectional=false, Parameters=58M x 2, Decoding=greedy2021.05 | 19.54 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=2-2-8, Layer-config=12-6, Compression-ratio=18.0x2022.03 | 17.66 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=2-2-8, Layer-config=12-3, Compression-ratio=19.9x2022.03 | 16.99 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=2-2-8, Layer-config=12-1, Compression-ratio=21.3x2022.03 | 12.81 | — | |
| DQ-mBART + DistillationProtocol=Distillation-Aware Quantization + Distillation, Bit-width=2-2-8, Layer-config=1-1, Compression-ratio=30.6x2022.03 | 10.36 | — | |
| mBART-large (direct quant.)Bit-width=8-8-8, Layer-config=12-12, Compression-ratio=4.0x2022.03 | 0.01 | — |