Natural Language Inference on AfriXNLI
90.8Accuracy (English)MERLIN
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MERLINModel Category=Key Multilingual Approaches, Base LLM=Gemma 2 9B, Encoder=NLLB-600M distilled2025.09 | 90.8 | 86.7 | 74.8 | 60 | 73.3 | 71.7 | 68.8 | 34.8 | 69.2 | 59.3 | 71.3 | 72 | 72.2 | 65.5 | 62.2 | 73.7 | 69.2 | 70.2 | 68.7 | |
| GPT-4oModel Category=closed models2025.09 | 89.2 | 82.3 | 71.8 | 45 | 75.2 | 68.2 | 68 | 32.7 | 69.8 | 71.2 | 71.3 | 71.8 | 71.5 | 55.8 | 52.7 | 72 | 64.5 | 67.5 | 64.3 | |
| GPT-4o-miniModel Category=closed models2025.09 | 86.2 | 80.3 | 52.2 | 37.2 | 64.8 | 57.2 | 56.8 | 31.5 | 51.7 | 61.3 | 63 | 56.2 | 63.7 | 47.5 | 43.2 | 64.5 | 54.5 | 62 | 54.2 | |
| QAlign-LoRAModel Category=Key Multilingual Approaches, Base LLM=Gemma 2 9B, Encoder=NLLB-600M distilled2025.09 | 81 | 74.7 | 51.3 | 35.7 | 55.8 | 49.6 | 43.3 | 34.3 | 41.5 | 39.7 | 48.5 | 48 | 60.7 | 43.8 | 35 | 51.5 | 47.5 | 49.3 | 49.5 | |
| MindMergerModel Category=Key Multilingual Approaches, Base LLM=Gemma 2 9B, Encoder=NLLB-600M distilled2025.09 | 80.8 | 76.3 | 69.2 | 60.5 | 70.8 | 68.5 | 66.8 | 34.5 | 70.3 | 54.7 | 75.2 | 71.2 | 71.3 | 63.7 | 59.5 | 73.8 | 68.8 | 74.3 | 67.1 | |
| Aya-101Model Category=open models2025.09 | 67 | 59.7 | 64.2 | 43.2 | 57 | 55.5 | 54.3 | 33.5 | 51.7 | 51.5 | 55.7 | 52.2 | 56.5 | 47 | 36.7 | 55.2 | 54.5 | 55.3 | 51.5 | |
| SLAMModel Category=Key Multilingual Approaches, Base LLM=Gemma 2 9B, Encoder=NLLB-600M distilled2025.09 | 67 | 61.8 | 48.2 | 42.7 | 47.2 | 51.3 | 41.7 | 32.2 | 47.5 | 47.7 | 46.5 | 47.2 | 52.8 | 38.8 | 35.7 | 49.8 | 52.2 | 48.3 | 47.7 | |
| Gemma 2 9BModel Category=open models2025.09 | 55.3 | 50.7 | 43.2 | 35.3 | 47 | 40.7 | 40.2 | 32.8 | 38.8 | 37.8 | 42.3 | 40.2 | 46.3 | 37.2 | 35.2 | 42.5 | 41.3 | 44.5 | 40.3 |