Label Projection on XQuAD
87.3COMET ScoreLabelPigeon
Evaluation results
| Method | Links | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LabelPigeonLanguage=Greek2026.03 | 87.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 75.8 | |
| LabelPigeonLanguage=Romanian2026.03 | 86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.8 | |
| LabelPigeonLanguage=Russian2026.03 | 85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.9 | |
| LabelPigeonLanguage=Turkish2026.03 | 85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 83.3 | |
| EasyProjectLanguage=Greek2026.03 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.8 | |
| Awesome-alignLanguage=Romanian2026.03 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 58 | |
| Awesome-alignLanguage=Greek2026.03 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 44.8 | |
| LabelPigeonLanguage=Spanish2026.03 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 90.2 | |
| Awesome-alignLanguage=Turkish2026.03 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 58.4 | |
| EasyProjectLanguage=Romanian2026.03 | 83.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.6 | |
| Gemma 3 27BLanguage=Romanian2026.03 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 89.2 | |
| Awesome-alignLanguage=Russian2026.03 | 83.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 52.5 | |
| Awesome-alignLanguage=Arabic2026.03 | 83.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49.8 | |
| LabelPigeonLanguage=Vietnamese2026.03 | 83.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 79.7 | |
| Awesome-alignLanguage=Spanish2026.03 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.2 | |
| Awesome-alignLanguage=Vietnamese2026.03 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 48.8 | |
| Awesome-alignLanguage=German2026.03 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60 | |
| EasyProjectLanguage=Turkish2026.03 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84 | |
| EasyProjectLanguage=Russian2026.03 | 82.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.6 | |
| LabelPigeonLanguage=German2026.03 | 82.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86.8 | |
| LabelPigeonLanguage=Average2026.03 | 82.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 79.2 | |
| Awesome-alignLanguage=Average2026.03 | 82.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 50.6 | |
| LabelPigeonLanguage=Arabic2026.03 | 82.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 75.3 | |
| EasyProjectLanguage=Spanish2026.03 | 81.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.4 | |
| EasyProjectLanguage=Arabic2026.03 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.9 | |
| EasyProjectLanguage=German2026.03 | 81.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84.6 | |
| Awesome-alignLanguage=Chinese2026.03 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 46.4 | |
| EasyProjectLanguage=Vietnamese2026.03 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.8 | |
| EasyProjectLanguage=Average2026.03 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77.7 | |
| EasyProjectLanguage=Chinese2026.03 | 79.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 70.9 | |
| Gemma 3 27BLanguage=Russian2026.03 | 79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.8 | |
| Gemma 3 27BLanguage=Turkish2026.03 | 78.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84.9 | |
| Awesome-alignLanguage=Hindi2026.03 | 78.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 54.8 | |
| LabelPigeonLanguage=Chinese2026.03 | 77.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.9 | |
| LabelPigeonLanguage=Hindi2026.03 | 77.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.9 | |
| Awesome-alignLanguage=Thai2026.03 | 76.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.8 | |
| LabelPigeonLanguage=Thai2026.03 | 76.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.1 | |
| EasyProjectLanguage=Hindi2026.03 | 76.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.6 | |
| EasyProjectLanguage=Thai2026.03 | 74.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 66 | |
| Gemma 3 27BLanguage=German2026.03 | 73.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86.4 | |
| Gemma 3 27BLanguage=Vietnamese2026.03 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.9 | |
| Gemma 3 27BLanguage=Greek2026.03 | 71.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 71.4 | |
| Gemma 3 27BLanguage=Average2026.03 | 69.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.1 | |
| Gemma 3 27BLanguage=Spanish2026.03 | 68.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.8 | |
| Gemma 3 27BLanguage=Thai2026.03 | 67.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 64.9 | |
| Gemma 3 27BLanguage=Chinese2026.03 | 64.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 70.6 | |
| Gemma 3 27BLanguage=Arabic2026.03 | 54.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 69.8 | |
| Gemma 3 27BLanguage=Hindi2026.03 | 50.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 71.3 | |
| Awesome-align2026.03 | — | 79.1 | 36 | 80.3 | 45 | 82.3 | 59.8 | 80.9 | 51.9 | 84.4 | 52 | 81.1 | 56.5 | 80.2 | 50.4 | 83.3 | 58 | 81.5 | 34.5 | 82.3 | 50.7 | 80.2 | 46.2 | 81.4 | 49.2 | — | |
| EasyProject2026.03 | — | 77.8 | 59.9 | 78.7 | 67.2 | 81.3 | 81 | 80.3 | 73.2 | 83.3 | 77.9 | 81.1 | 78.6 | 79.1 | 75.1 | 82.7 | 83.1 | 79.2 | 59.7 | 82.1 | 75 | 80.2 | 74.8 | 80.5 | 73.2 | — | |
| Gemma 3 27BBackbone=Gemma 3, Number of parameters=27B2026.03 | — | 81.6 | 69.4 | 81.9 | 64.7 | 85.2 | 80.3 | 84.9 | 73.4 | 84.8 | 74.2 | 85.7 | 85.6 | 83.1 | 72 | 85.3 | 79.3 | 82.6 | 64.6 | 85.9 | 82 | 84.5 | 82.2 | 84.1 | 75.3 | — | |
| LabelPigeon2026.03 | — | 80.6 | 76 | 82.7 | 80 | 83.8 | 85.6 | 81.7 | 77.3 | 85.3 | 83.9 | 83 | 82.6 | 81.6 | 80.7 | 85.1 | 87.1 | 80.3 | 70.9 | 84.8 | 82.9 | 83.5 | 79.9 | 82.9 | 80.6 | — |