Cross-lingual Language Understanding on XTREME
83.9XNLI AccuracyFILTER
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FILTER# PT params=559M, # FT params=559M, Fine-tuning strategy=fine-tuned on translations or additional task data2020.10 | 83.9 | 76.2 | 67.7 | 91.4 | 68 | 82.4 | 57.7 | 76.2 | 50.9 | 68.3 | 74.9 | 76 | — | — | — | |
| VECO# PT params=662M, # FT params=662M, Fine-tuning strategy=fine-tuned on translations or additional task data2020.10 | 83 | 75.1 | 65.7 | 91.1 | 66.3 | 79.9 | 54.9 | 73.1 | 58.9 | 75 | 74.1 | 75.1 | — | — | — | |
| XLM-Rlarge+NoisyTuneProtocol=Translate-Train-All2022.02 | 82.5 | 77.8 | 67.9 | 90.5 | 66.1 | 80.4 | 54.9 | 73.3 | 48.2 | 66.8 | — | 76.8 | 82.9 | 83 | — | |
| XLM-RlargeProtocol=Translate-Train-All2022.02 | 82.3 | 77.3 | 67.3 | 90.3 | 65.6 | 80 | 54.4 | 72.9 | 47.6 | 66.3 | — | 76.4 | 82.5 | 82.7 | — | |
| InfoXLM_largeModel Size=large, Zero-shot transfer=true2023.06 | 81.4 | — | — | — | — | — | 55.2 | 73.6 | — | — | — | — | — | — | — | |
| MrBERTNumber of Parameters=308M, Fine-tuned on English data=true2026.02 | 81.26 | 83.74 | — | 91.32 | — | 77.91 | — | 70.67 | — | 56.34 | 76.19 | — | — | — | 72.06 | |
| XLM-P_largeFine-tuning protocol=Prompt-based, Model Size=large, Zero-shot transfer=true2023.06 | 81.2 | 77 | 68.5 | 88.9 | 64.4 | 79.2 | 56.4 | 73.7 | 55.7 | 72.7 | — | — | — | — | — | |
| XLM-P_largeFine-tuning protocol=Standard, Model Size=large, Zero-shot transfer=true2023.06 | 81.1 | 76.9 | 68.1 | 88.7 | 63.9 | 79 | 53.5 | 72.4 | 55 | 72.1 | — | — | — | — | — | |
| RemBERT# PT params=995M, # FT params=575M, Fine-tuning strategy=fine-tuned only on English task data2020.10 | 80.8 | 76.5 | 70.1 | 87.5 | 64 | 79.6 | 55 | 73.1 | 63 | 77 | 74.4 | 75.4 | — | — | — | |
| mmBERTNumber of Parameters=308M, Fine-tuned on English data=true2026.02 | 80.54 | 84.33 | — | 92.34 | — | 77.79 | — | 71.48 | — | 63.95 | 77.76 | — | — | — | 73.89 | |
| STiLTs# PT params=559M, # FT params=559M, Fine-tuning strategy=fine-tuned on translations or additional task data2020.10 | 80 | 74.9 | 64 | 87.9 | 63.3 | 78.7 | 53.7 | 72.4 | 59.5 | 76 | 72.7 | 73.5 | — | — | — | |
| VECO_largeModel Size=large, Zero-shot transfer=true2023.06 | 79.9 | 75.1 | 65.7 | 88.7 | 61.8 | 77.3 | 53.2 | 71.7 | 49.1 | 67.6 | — | — | — | — | — | |
| XLM-Rlarge+NoisyTuneProtocol=Cross-lingual Transfer2022.02 | 79.3 | 73.5 | 63.2 | 86.5 | 61 | 76.7 | 53.6 | 71.9 | 45.6 | 65.4 | — | 73 | 79.9 | 76.8 | — | |
| XLM-R# PT params=559M, # FT params=559M, Fine-tuning strategy=fine-tuned only on English task data2020.10 | 79.2 | 73.8 | 65.4 | 86.4 | 60.8 | 76.6 | 53.2 | 71.6 | 45 | 65.1 | 70.1 | 71.4 | — | — | — | |
| XLM-R_largeModel Size=large, Zero-shot transfer=true2023.06 | 79.2 | 72.6 | 65.4 | 86.4 | 60.8 | 76.6 | 53.2 | 71.6 | 45 | 65.1 | — | — | — | — | — | |
| mRoBERTaNumber of Parameters=283M, Fine-tuned on English data=true2026.02 | 79.09 | 85.36 | — | 90.36 | — | 75.45 | — | 68.67 | — | 53.96 | 75.51 | — | — | — | 75.65 | |
| XLM-RlargeProtocol=Cross-lingual Transfer2022.02 | 79 | 72.7 | 62.3 | 86.3 | 60.4 | 76.2 | 53 | 71.4 | 45 | 65 | — | 72.4 | 79.2 | 76 | — | |
| XLM-Rbase+NoisyTuneProtocol=Translate-Train-All2022.02 | 78.9 | 76.8 | 63.1 | 88.6 | 61.8 | 75.4 | 50.4 | 68 | 48.1 | 64.1 | — | 73.7 | 80 | 79.8 | — | |
| XLM-RbaseProtocol=Translate-Train-All2022.02 | 78.5 | 76.2 | 62.6 | 88.2 | 61.5 | 75 | 50.1 | 67.8 | 47.6 | 63.8 | — | 73.3 | 79.6 | 79.4 | — | |
| xlm-roberta-baseNumber of Parameters=279M, Fine-tuned on English data=true2026.02 | 78.25 | 85.55 | — | 89.5 | — | 75.61 | — | 68.91 | — | 56.41 | 75.42 | — | — | — | 73.69 | |
| mGTENumber of Parameters=306M, Fine-tuned on English data=true2026.02 | 77.9 | 82.5 | — | 89.55 | — | 74.37 | — | 68.05 | — | 51.07 | 73.78 | — | — | — | 73.05 | |
| XLM-EPre-training Corpus=multilingual and parallel corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer, Number of Runs=Averaged over 5 runs2021.06 | 76.6 | 75.6 | 63.5 | 88.3 | 60.2 | 76.2 | 49.8 | 68.3 | 45.7 | 62.4 | 69.3 | — | — | — | — | |
| INFOXLMPre-training Corpus=multilingual and parallel corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 76.5 | — | — | — | — | — | 49.6 | 68.1 | — | — | — | — | — | — | — | |
| XLM-ALIGNPre-training Corpus=multilingual and parallel corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 76.2 | 76 | 63.7 | 86.8 | 59 | 74.7 | 49.8 | 68.1 | 44.8 | 62.1 | 68.9 | — | — | — | — | |
| XLM-ALIGNSetting=Cross-lingual transfer, Fine-tuning=English only, Average=5 runs2021.06 | 76.2 | 76 | 63.7 | 86.8 | 59 | 74.7 | 49.8 | 68.1 | 44.8 | 62.1 | 68.9 | — | — | — | — | |
| MT5Pre-training Corpus=multilingual corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 75.4 | — | 55.7 | 86.4 | 49 | 67 | 45 | 64.6 | 41.2 | 57.2 | — | — | — | — | — | |
| XLM-P_baseFine-tuning protocol=Standard, Model Size=base, Zero-shot transfer=true2023.06 | 75.4 | 74.3 | 63.8 | 86.6 | 60.4 | 75.3 | 49.4 | 67.4 | 41.6 | 58.5 | — | — | — | — | — | |
| MT5baseSetting=Cross-lingual transfer, Fine-tuning=English only2021.06 | 75.4 | — | 56.6 | 87.4 | 49 | 67 | 45 | 64.6 | 42.8 | 58.1 | — | — | — | — | — | |
| XLM-Rbase+NoisyTuneProtocol=Cross-lingual Transfer2022.02 | 75.2 | 76 | 62.1 | 85.1 | 57.1 | 72.3 | 47.4 | 65.5 | 39.2 | 56 | — | 70.5 | 78.2 | 74.5 | — | |
| XLM-E (w/o TRTD)Pre-training Corpus=multilingual corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 75.1 | 74.2 | 62.7 | 87.1 | 58.2 | 74.3 | 49.7 | 67.8 | 40.6 | 57.8 | 67.6 | — | — | — | — | |
| MT5_baseModel Size=base, Zero-shot transfer=true2023.06 | 75.1 | — | 55.7 | 86.4 | 49 | 67 | 45 | 64.6 | 41.2 | 57.2 | — | — | — | — | — | |
| XLM-E (-TRTD)_baseModel Size=base, Zero-shot transfer=true2023.06 | 75.1 | 74.2 | 62.7 | 87.1 | 58.2 | 74.3 | 49.7 | 67.8 | 40.6 | 57.8 | — | — | — | — | — | |
| XLM-P_baseFine-tuning protocol=Prompt-based, Model Size=base, Zero-shot transfer=true2023.06 | 75.1 | 73.9 | 63.6 | 86 | 61.5 | 75.8 | 50.1 | 68.7 | 42.8 | 59.3 | — | — | — | — | — | |
| XLM-RPre-training Corpus=multilingual corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer, Number of Runs=Averaged over 5 runs2021.06 | 75 | 75.6 | 61.8 | 84.9 | 56.4 | 71.9 | 47.2 | 65.1 | 38.3 | 55.4 | 66.4 | — | — | — | — | |
| XLM-R_baseModel Size=base, Zero-shot transfer=true2023.06 | 75 | 75.6 | 61.8 | 84.9 | 56.4 | 71.9 | 47.2 | 65.1 | 38.3 | 55.4 | — | — | — | — | — | |
| XLM-RbaseSetting=Cross-lingual transfer, Fine-tuning=English only, Average=5 runs2021.06 | 75 | 75.6 | 61.8 | 84.9 | 56.4 | 71.9 | 47.2 | 65.1 | 38.3 | 55.4 | 66.4 | — | — | — | — | |
| XLM-RbaseProtocol=Cross-lingual Transfer2022.02 | 74.8 | 75.5 | 61.6 | 84.8 | 56.6 | 71.9 | 47 | 65.2 | 38.4 | 55.5 | — | 70 | 77.6 | 73.8 | — | |
| XLMPre-training Corpus=multilingual and parallel corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 69.1 | 70.1 | 61.2 | 80.9 | 44.3 | 59.8 | 32.6 | 48.5 | 29.1 | 43.6 | 58.6 | — | — | — | — | |
| XLM_baseModel Size=base, Zero-shot transfer=true2023.06 | 69.1 | 71.3 | 61.2 | 80.9 | 44.3 | 59.8 | 32.6 | 48.5 | 29.1 | 43.6 | — | — | — | — | — | |
| XLM*Setting=Cross-lingual transfer, Fine-tuning=English only, Source=Hu et al., 2020b2021.06 | 69.1 | 70.1 | 61.2 | 80.9 | 44.3 | 59.8 | 32.6 | 48.5 | 29.1 | 43.6 | 58.6 | — | — | — | — | |
| MBERT*Setting=Cross-lingual transfer, Fine-tuning=English only, Source=Hu et al., 2020b2021.06 | 65.4 | 70.3 | 62.2 | 81.9 | 49.4 | 64.5 | 44.2 | 61.4 | 43.9 | 59.7 | 63.1 | — | — | — | — | |
| MBERTPre-training Corpus=multilingual corpus, Model Size=Base, Evaluation Protocol=Cross-lingual transfer2021.06 | 65.1 | 70.3 | 62.2 | 81.9 | 49.4 | 64.5 | 44.2 | 61.4 | 43.9 | 59.7 | 63.1 | — | — | — | — | |
| MBERT_baseModel Size=base, Zero-shot transfer=true2023.06 | 65.1 | 70.3 | 62.2 | 81.9 | 49.4 | 64.5 | 44.2 | 61.4 | 43.9 | 59.7 | — | — | — | — | — | |
| mBERTParams=167M, FLOPs=6.4e192021.06 | — | — | — | — | — | — | — | — | — | — | — | 63.1 | — | — | — | |
| XLM-ALIGNParams=279M, FLOPs=9.6e21 + 9.6e19, continue-trained from XLM-R=true2021.06 | — | — | — | — | — | — | — | — | — | — | — | 68.9 | — | — | — | |
| XLM-EParams=279M, FLOPs=9.5e192021.06 | — | — | — | — | — | — | — | — | — | — | — | 69.3 | — | — | — | |
| XLM-E (w/o TRTD)Params=279M, FLOPs=6.3e192021.06 | — | — | — | — | — | — | — | — | — | — | — | 67.6 | — | — | — | |
| XLM-RParams=279M, FLOPs=9.6e212021.06 | — | — | — | — | — | — | — | — | — | — | — | 66.4 | — | — | — |