Question Answering on MLQA (test)
76.2F1 ScoreFILTER
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FILTERProtocol=Translate-train, Learning approach=Self-Teaching2020.09 | 76.2 | 57.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-ESize=XL, Params=2.2B2021.06 | 76.2 | 57.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FILTERProtocol=Translate-train2020.09 | 75.8 | 57.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfoXLMProtocol=Cross-lingual zero-shot transfer2020.09 | 73.6 | 55.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT5Size=XL, Params=3.7B2021.06 | 73.5 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-RSize=XL, Params=3.5B2021.06 | 73.4 | 55.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-RProtocol=Translate-train, Learning approach=multi-task2020.09 | 72.8 | 54.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-ESize=Large, Params=840M2021.06 | 72.7 | 54.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X-STILTSProtocol=Cross-lingual zero-shot transfer2020.09 | 72.4 | 53.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-RProtocol=Cross-lingual zero-shot transfer2020.09 | 71.6 | 53.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLM-ESize=Base, Params=279M2021.06 | 68.3 | 49.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mBERTProtocol=Translate-train, Learning approach=multi-task2020.09 | 67.6 | 49.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our methodM (few-shot samples)=202023.05 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning slow algorithm (FS)M (few-shot samples)=202023.05 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our methodM (few-shot samples)=52023.05 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our methodM (few-shot samples)=102023.05 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning fast algorithm (FF)M (few-shot samples)=202023.05 | 66.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our methodM (few-shot samples)=02023.05 | 66.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning fast algorithm (FF)M (few-shot samples)=52023.05 | 66.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning fast algorithm (FF)M (few-shot samples)=102023.05 | 66.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning slow algorithm (FS)M (few-shot samples)=102023.05 | 66.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning fast algorithm (FF)M (few-shot samples)=02023.05 | 66.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning slow algorithm (FS)M (few-shot samples)=52023.05 | 66.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuning slow algorithm (FS)M (few-shot samples)=02023.05 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mBERTProtocol=Translate-train2020.09 | 65.6 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NoisyTuneM (few-shot samples)=202023.05 | 65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuning (DF)M (few-shot samples)=202023.05 | 64.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NoisyTuneM (few-shot samples)=102023.05 | 64.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuning (DF)M (few-shot samples)=52023.05 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuning (DF)M (few-shot samples)=102023.05 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NoisyTuneM (few-shot samples)=02023.05 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NoisyTuneM (few-shot samples)=52023.05 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuning (DF)M (few-shot samples)=02023.05 | 64.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mBERTProtocol=Cross-lingual zero-shot transfer2020.09 | 61.4 | 44.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLMProtocol=Cross-lingual zero-shot transfer2020.09 | 48.5 | 32.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Lewis et al.mode=zero-shot2021.06 | — | — | 77.7 | 45.7 | 57.9 | 64.3 | 43.8 | 57.1 | 57.5 | 57.7 | 65.2 | 46.6 | 44.3 | 29.8 | 29.7 | 38.6 | 37.3 | 41.6 | |
| mBERTbaseline=true2021.06 | — | — | 80.5 | 47.2 | 59 | 63.9 | 47.5 | 56.5 | 56.6 | 58.7 | — | — | — | — | — | — | — | — | |
| mBERT (Reference [3])Reference=[3]2021.06 | — | — | 77.7 | 45.7 | 57.9 | 64.3 | 43.8 | 57.1 | 57.5 | 57.7 | — | — | — | — | — | — | — | — | |
| mBERT (reproduced)mode=zero-shot2021.06 | — | — | 80.5 | 47.2 | 59 | 63.9 | 47.5 | 56.5 | 57.8 | 58.9 | 67.2 | 44.1 | 43.3 | 28.6 | 32.1 | 35.2 | 33 | 40.5 | |
| mBERT+Syn. (Syntax-augmented mBERT)mode=zero-shot2021.06 | — | — | 80.4 | 48.9 | 60.8 | 65.9 | 46.7 | 59.3 | 60.1 | 60.3 | 67.3 | 47.1 | 45.1 | 30.3 | 32.4 | 38 | 35.5 | 42.2 | |
| Syntax-augmented mBERTsyntax-augmented=true2021.06 | — | — | 80.4 | 48.9 | 60.8 | 65.9 | 46.7 | 59.3 | 60.1 | 60.3 | — | — | — | — | — | — | — | — |