Multilingual Knowledge Evaluation on M-MMLU
77.83Overall AccuracyKarnak
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| KarnakSize=40B, Protocol=0-shot2026.03 | 77.83 | — | — | — | — | — | — | — | — | |
| CAUSAL-MULTIModel=GPT-4o2025.05 | 73.8 | 76.5 | 83 | 76 | 79.5 | 74 | 66 | 67.5 | 67.1 | |
| MULTI-RELATEDModel=GPT-4o2025.05 | 72.5 | 77 | 85 | 74 | 80 | 72 | 60.5 | 67.5 | 63.9 | |
| CAUSAL-NATIVEModel=GPT-4o2025.05 | 71.3 | 80.5 | 75.6 | 76 | 80.5 | 73.7 | 61 | 60.8 | 62.3 | |
| Llama-3.3-70b-InstructSize=70B, Protocol=0-shot2026.03 | 70 | — | — | — | — | — | — | — | — | |
| Qwen3-32BSize=32B, Protocol=0-shot2026.03 | 69.32 | — | — | — | — | — | — | — | — | |
| Jais-2-70B-ChatSize=70B, Protocol=0-shot2026.03 | 69.01 | — | — | — | — | — | — | — | — | |
| MOREINFOModel=GPT-4o2025.05 | 68.6 | 76.8 | 83.3 | 68.2 | 77.9 | 61.3 | 59.7 | 66.9 | 54.3 | |
| AceGPT-v2-70B-ChatSize=70B, Protocol=0-shot2026.03 | 68.44 | — | — | — | — | — | — | — | — | |
| Gemma-3-27B-itSize=27B, Protocol=0-shot2026.03 | 67.65 | — | — | — | — | — | — | — | — | |
| Fanar-27BSize=27B, Protocol=0-shot2026.03 | 67.4 | — | — | — | — | — | — | — | — | |
| AceGPT-v2-32B-ChatSize=32B, Protocol=0-shot2026.03 | 61.1 | — | — | — | — | — | — | — | — | |
| CONFLICTModel=GPT-4o2025.05 | 59.9 | 66.5 | 69.2 | 62.3 | 73.4 | 50.3 | 51.9 | 48.7 | 56.5 | |
| Fanar-1-9b-instructSize=9B, Protocol=0-shot2026.03 | 58.3 | — | — | — | — | — | — | — | — | |
| Allam-7B-Instruct-preview-v2Size=7B, Protocol=0-shot2026.03 | 57.16 | — | — | — | — | — | — | — | — | |
| CAUSAL-MULTIModel=ChatGPT2025.05 | 56.7 | 57.5 | 62 | 52.3 | 59.5 | 57 | 58.1 | 56.8 | 50 | |
| CONFLICTModel=ChatGPT2025.05 | 55.3 | 52.2 | 53.4 | 52.2 | 53 | 60.8 | 56.2 | 59.9 | 54.9 | |
| MULTI-RELATEDModel=AYA-13B2025.05 | 54.9 | 55.5 | 52.5 | 56 | 51 | 55.5 | 52.4 | 55.5 | 60.9 | |
| CAUSAL-MULTIModel=AYA-13B2025.05 | 54.5 | 51.2 | 53.8 | 53 | 53.5 | 57.4 | 54.4 | 54.5 | 57.9 | |
| ASK CALIModel=AYA-13B2025.05 | 53.2 | 55.2 | 53.4 | 59.3 | 54.7 | 58.7 | 42.4 | 47.7 | 53.9 | |
| CAUSAL-NATIVEModel=ChatGPT2025.05 | 53.2 | 57 | 59 | 48 | 60 | 49.3 | 56.5 | 45.5 | 50.3 | |
| REFLECTModel=ChatGPT2025.05 | 52.8 | 54.3 | 45.7 | 47 | 56.6 | 51.6 | 58.8 | 55 | 53.4 | |
| ASK CALIModel=ChatGPT2025.05 | 52.2 | 44 | 38.5 | 53.5 | 42.7 | 61.8 | 62.2 | 56.8 | 57.5 | |
| MULTI-RELATEDModel=ChatGPT2025.05 | 52.1 | 56.5 | 62.5 | 52 | 57 | 50.5 | 49 | 48.5 | 40.4 | |
| CONFLICTModel=AYA-13B2025.05 | 50.4 | 47 | 54.7 | 45.6 | 46.1 | 55 | 49.7 | 55 | 50 | |
| CAUSAL-NATIVEModel=AYA-13B2025.05 | 50.4 | 48.2 | 53.8 | 51 | 47.5 | 55 | 49.7 | 47.5 | 50.7 | |
| REFLECTModel=GPT-4o2025.05 | 50.4 | 50.3 | 52.6 | 50.3 | 55.2 | 46.2 | 46.5 | 49.3 | 52.7 | |
| MOREINFOModel=ChatGPT2025.05 | 44.6 | 56 | 61.1 | 45.1 | 53.4 | 30 | 40.1 | 32 | 39 | |
| MOREINFOModel=AYA-13B2025.05 | 43.1 | 50 | 47.9 | 50 | 43.5 | 34.8 | 42.9 | 35.3 | 41.1 | |
| REFLECTModel=AYA-13B2025.05 | 42.3 | 47.8 | 47.4 | 46.9 | 46.3 | 37 | 39.5 | 32 | 41.8 | |
| ASK CALIModel=GPT-4o2025.05 | 34.3 | 20.8 | 20.5 | 29.3 | 20.8 | 46.2 | 41.3 | 41.9 | 53.1 |