Dialectal Arabic Evaluation on MMLU Dialectal Arabic
69.98Accuracy (MMLU DA)Karnak
Evaluation results
| Method | Links | |
|---|---|---|
| KarnakSize=40B, Shot count=3-shot2026.03 | 69.98 | |
| Llama-3.3-70b-InstructSize=70B, Shot count=3-shot2026.03 | 69.63 | |
| Qwen3-32BSize=32B, Shot count=3-shot2026.03 | 68.96 | |
| AceGPT-v2-70B-ChatSize=70B, Shot count=3-shot2026.03 | 68.56 | |
| Fanar-27BSize=27B, Shot count=3-shot2026.03 | 67.4 | |
| Gemma-3-27B-itSize=27B, Shot count=3-shot2026.03 | 66.52 | |
| Jais-2-70B-ChatSize=70B, Shot count=3-shot2026.03 | 66.33 | |
| AceGPT-v2-32B-ChatSize=32B, Shot count=3-shot2026.03 | 62.82 | |
| Fanar-1-9b-instructSize=9B, Shot count=3-shot2026.03 | 59.91 | |
| Allam-7B-Instruct-preview-v2Size=7B, Shot count=3-shot2026.03 | 56.17 |