Linguistic Acceptability on URBLIMP
100Aspect AccuracyDunbaaBERT32k
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DunbaaBERT32kVocabulary Size=32k2026.05 | 100 | 95.6 | 99.2 | 84.5 | 99.6 | 88.5 | 95.8 | 85.6 | 99 | 95 | 98.5 | 98 | 93.2 | 98 | 95.7 | 97.8 | 98.5 | 91.1 | 93.1 | — | 95.1 | |
| DunbaaBERT52kVocabulary Size=52k2026.05 | 100 | 92.5 | 98.5 | 95.6 | 99.8 | 90 | 98.3 | 90.8 | 99.5 | 94 | 99 | 98 | 96.1 | 98 | 97.3 | 99.3 | 98.5 | 99 | 98 | — | 97 | |
| Urdu-RoBERTasmallModel Size=small2026.05 | 99.8 | 86.9 | 90.9 | 96 | 99.6 | 86.5 | 95.8 | 71.2 | 92.5 | 89 | 99 | 84.3 | 84.5 | 99 | 89.7 | 81.6 | 89.8 | 95.1 | 88.1 | — | 90.5 | |
| XLM-RbaseModel Size=base2026.05 | 99.5 | 93.1 | 94.7 | 96.9 | 99.3 | 72.5 | 92.8 | 61.4 | 88.5 | 89 | 99 | 91.2 | 89.3 | 94.1 | 79.4 | 84.6 | 88.3 | 90.2 | 98 | — | 89.6 | |
| mmBERTsmallModel Size=small2026.05 | 99.3 | 88.1 | 96.2 | 93.3 | 98.4 | 82 | 73.8 | 65.4 | 87 | 91 | 93.3 | 95.1 | 84.5 | 96.1 | 89.7 | 85.4 | 91.3 | 92.1 | 99 | — | 89.5 | |
| XLM-RlargeModel Size=large2026.05 | 99.3 | 95 | 97.7 | 90 | 99.8 | 92 | 95.3 | 79.7 | 93.5 | 92 | 98.4 | 92.2 | 88.3 | 97.1 | 92 | 93.6 | 96.1 | 98.7 | 98 | — | 94.1 | |
| mBERT2026.05 | 98.8 | 65 | 53 | 85.1 | 95.7 | 63 | 18.5 | 58.2 | 86.5 | 77 | 93.7 | 74.5 | 78.6 | 95.1 | 82.4 | 73.8 | 88.3 | 55.4 | 91.1 | — | 75.5 | |
| DunbaaBERT96kVocabulary Size=96k2026.05 | 98.5 | 95.6 | 95.5 | 94.5 | 98.7 | 91 | 90.6 | 92.8 | 97 | 88 | 97.2 | 95.1 | 94.2 | 96.1 | 97 | 93.3 | 97.6 | 98 | 87.1 | 87.1 | 94.6 | |
| HPLT-BERTur2026.05 | 98.3 | 98.8 | 99.2 | 98.4 | 99.9 | 97 | 96.5 | 94.8 | 97.5 | 95 | 99.2 | 95.1 | 95.1 | 96.1 | 98 | 98.1 | 97.1 | 97 | 98 | — | 97.3 | |
| mmBERTbaseModel Size=base2026.05 | 98 | 90 | 97.7 | 94.5 | 99.6 | 89 | 89.4 | 71.2 | 94 | 92 | 97.6 | 95.1 | 85.4 | 94.1 | 92 | 91.4 | 91.7 | 95.7 | 97 | — | 92.4 |