Machine Reading Comprehension on SQuAD (EM, F1)
89.9EMExpanded Vocab
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Expanded VocabTraining Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 89.9 | 95.7 | |
| Base Llama2Training Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 89.5 | 95.3 | |
| Token-Opt Trans.Training Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 88 | 89.3 | |
| Basic Trans.Training Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 87.7 | 89.1 | |
| Hybrid Trans.Training Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 87.2 | 89 | |
| UROMANTraining Setting=SQuAD-Trained, Evaluation Protocol=Source Language Fine-tuning2025.09 | 84.2 | 88.7 | |
| Llama 2Size=70B, shot=1-shot2023.07 | 82.6 | — | |
| Llama 2Size=70B, shot=4-shot2023.07 | 81.9 | — | |
| Llama 2Size=70B, shot=5-shot2023.07 | 81.9 | — | |
| Llama 2Size=70B, shot=0-shot2023.07 | 80.7 | — | |
| Llama 1Size=65B, shot=1-shot2023.07 | 80 | — | |
| Llama 1Size=65B, shot=0-shot2023.07 | 79.4 | — | |
| Llama 2Size=34B, shot=1-shot2023.07 | 78.8 | — | |
| Llama 1Size=65B, shot=4-shot2023.07 | 78.3 | — | |
| Llama 1Size=65B, shot=5-shot2023.07 | 77.9 | — | |
| Llama 2Size=34B, shot=4-shot2023.07 | 77.5 | — | |
| Llama 2Size=34B, shot=5-shot2023.07 | 77.5 | — | |
| Llama 2Size=34B, shot=0-shot2023.07 | 77.4 | — | |
| Llama 1Size=33B, shot=1-shot2023.07 | 77 | — | |
| Llama 1Size=33B, shot=4-shot2023.07 | 76.3 | — | |
| Llama 1Size=33B, shot=5-shot2023.07 | 75.6 | — | |
| Llama 1Size=33B, shot=0-shot2023.07 | 75.5 | — | |
| MPTSize=30B, shot=0-shot2023.07 | 74.7 | — | |
| MPTSize=30B, shot=1-shot2023.07 | 74.2 | — | |
| MPTSize=30B, shot=5-shot2023.07 | 74.2 | — | |
| FalconSize=40B, shot=1-shot2023.07 | 73.1 | — | |
| FalconSize=40B, shot=0-shot2023.07 | 72.9 | — | |
| Llama 2Size=13B, shot=0-shot2023.07 | 72.9 | — | |
| Llama 2Size=7B, shot=4-shot2023.07 | 72.6 | — | |
| Llama 2Size=7B, shot=5-shot2023.07 | 72.5 | — | |
| MPTSize=30B, shot=4-shot2023.07 | 72.4 | — | |
| Llama 2Size=7B, shot=1-shot2023.07 | 72.3 | — | |
| Llama 2Size=13B, shot=1-shot2023.07 | 72.1 | — | |
| FalconSize=40B, shot=4-shot2023.07 | 71.7 | — | |
| Llama 2Size=13B, shot=5-shot2023.07 | 71.3 | — | |
| FalconSize=40B, shot=5-shot2023.07 | 71 | — | |
| Llama 2Size=13B, shot=4-shot2023.07 | 70.6 | — | |
| Llama 1Size=13B, shot=0-shot2023.07 | 68.9 | — | |
| Llama 1Size=13B, shot=1-shot2023.07 | 68.4 | — | |
| Llama 2Size=7B, shot=0-shot2023.07 | 67.2 | — | |
| Llama 1Size=13B, shot=5-shot2023.07 | 66.7 | — | |
| Llama 1Size=13B, shot=4-shot2023.07 | 66.4 | — | |
| Llama 1Size=7B, shot=4-shot2023.07 | 63.3 | — | |
| MPTSize=7B, shot=1-shot2023.07 | 62.8 | — | |
| Llama 1Size=7B, shot=5-shot2023.07 | 62.8 | — | |
| MPTSize=7B, shot=5-shot2023.07 | 62.7 | — | |
| MPTSize=7B, shot=4-shot2023.07 | 62.6 | — | |
| Llama 1Size=7B, shot=1-shot2023.07 | 62.3 | — | |
| Llama 1Size=7B, shot=0-shot2023.07 | 60 | — | |
| MPTSize=7B, shot=0-shot2023.07 | 59.5 | — | |
| RandomBackbone=Llama-2-7B-Chat, Noise Type=Clean, Noise Ratio=0%2024.05 | 56.5 | — | |
| DPP + LPRBackbone=Llama-2-7B-Chat, Noise Type=Relevant Noise, Noise Ratio=60%2024.05 | 50.77 | — | |
| TopK + LPRBackbone=Llama-2-7B-Chat, Noise Type=Relevant Noise, Noise Ratio=60%2024.05 | 48.37 | — | |
| Random + LPRBackbone=Llama-2-7B-Chat, Noise Type=Irrelevant Noise, Noise Ratio=60%2024.05 | 43 | — | |
| FalconSize=7B, shot=5-shot2023.07 | 17.5 | — | |
| FalconSize=7B, shot=4-shot2023.07 | 16.9 | — | |
| FalconSize=7B, shot=0-shot2023.07 | 16.4 | — | |
| FalconSize=7B, shot=1-shot2023.07 | 16 | — |