Multilingual Language Understanding on M-MMLU (test)
59.6Overall AccuracyCOMPASS
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| COMPASSBackbone=Qwen2.5-7B-Instruct2026.04 | 59.6 | — | — | — | — | — | — | — | — | |
| COMPASS-FFTBackbone=Qwen2.5-7B-Instruct, Fine-tuning=Full2026.04 | 58.1 | — | — | — | — | — | — | — | — | |
| LangRankBackbone=Qwen2.5-7B-Instruct2026.04 | 56.7 | — | — | — | — | — | — | — | — | |
| CAUSAL-MULTIBackbone=LLaMa 3.22025.05 | 56.5 | 61.2 | 58.9 | 57.6 | 52.3 | 49.6 | 58.1 | 58.1 | 55.8 | |
| LangSimBackbone=Qwen2.5-7B-Instruct2026.04 | 56.1 | — | — | — | — | — | — | — | — | |
| COMPASS-FFTBackbone=Llama-3.1-Instruct-8B, Fine-tuning=Full2026.04 | 55.5 | — | — | — | — | — | — | — | — | |
| COMPASSBackbone=Llama-3.1-Instruct-8B2026.04 | 55.2 | — | — | — | — | — | — | — | — | |
| RandomBackbone=Qwen2.5-7B-Instruct2026.04 | 55.1 | — | — | — | — | — | — | — | — | |
| CAUSAL-MULTIBackbone=Phi 42025.05 | 54.7 | 57.4 | 64.7 | 60.9 | 58.4 | 48.2 | 44.1 | 50.6 | 53.1 | |
| TargetBackbone=Qwen2.5-7B-Instruct2026.04 | 54.6 | — | — | — | — | — | — | — | — | |
| REFLECTBackbone=Phi 42025.05 | 53.1 | 65.2 | 60.3 | 57 | 57.8 | 45.5 | 45 | 48 | 45.7 | |
| Pretrained (ZS)Backbone=Qwen2.5-7B-Instruct, Evaluation Protocol=Zero-shot2026.04 | 52.9 | — | — | — | — | — | — | — | — | |
| COMPASSBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 52.4 | — | — | — | — | — | — | — | — | |
| CAUSAL-NATIVEBackbone=LLaMa 3.22025.05 | 52.2 | 57.4 | 47.4 | 53.6 | 53.2 | 45.5 | 48 | 58 | 54.2 | |
| LangRankBackbone=Llama-3.1-Instruct-8B2026.04 | 51.8 | — | — | — | — | — | — | — | — | |
| CONFLICTBackbone=Phi 42025.05 | 51.4 | 56.8 | 52.6 | 45.7 | 43.5 | 47.6 | 50.4 | 52.7 | 52.7 | |
| MULTI-RELATEDBackbone=Phi 42025.05 | 51.4 | 49.7 | 53.2 | 56.3 | 45.5 | 49.7 | 44.2 | 58.1 | 54.3 | |
| REFLECTBackbone=LLaMa 3.22025.05 | 51.2 | 41.9 | 51.9 | 57 | 54.6 | 53.1 | 50.4 | 48 | 52.7 | |
| CAUSAL-NATIVEBackbone=Phi 42025.05 | 51.2 | 51.6 | 60.8 | 58.9 | 58.4 | 44.1 | 41 | 44.5 | 50 | |
| LangSimBackbone=Llama-3.1-Instruct-8B2026.04 | 51 | — | — | — | — | — | — | — | — | |
| TargetBackbone=Llama-3.1-Instruct-8B2026.04 | 50.8 | — | — | — | — | — | — | — | — | |
| RandomBackbone=Llama-3.1-Instruct-8B2026.04 | 50.6 | — | — | — | — | — | — | — | — | |
| MOREINFOBackbone=Phi 42025.05 | 50.5 | 48.4 | 46.8 | 54.3 | 48.7 | 40.7 | 54.3 | 60.1 | 50.4 | |
| COMPASS-FFTBackbone=Phi-4-Mini-Instruct-3.8B, Fine-tuning=Full2026.04 | 49.9 | — | — | — | — | — | — | — | — | |
| CONFLICTBackbone=LLaMa 3.22025.05 | 49.6 | 51 | 53.2 | 53.6 | 49.4 | 42.1 | 52.7 | 51.4 | 43.4 | |
| Pretrained (ZS)Backbone=Llama-3.1-Instruct-8B, Evaluation Protocol=Zero-shot2026.04 | 49.1 | — | — | — | — | — | — | — | — | |
| ASK CALIBackbone=LLaMa 3.22025.05 | 48.5 | 64.6 | 44.2 | 69.5 | 66.2 | 26.9 | 38.8 | 30.4 | 47.3 | |
| MOREINFOBackbone=LLaMa 3.22025.05 | 48.4 | 53.6 | 50.6 | 49 | 47.4 | 49.7 | 38 | 51.4 | 47.3 | |
| AllBackbone=Qwen2.5-7B-Instruct2026.04 | 48 | — | — | — | — | — | — | — | — | |
| MULTI-RELATEDBackbone=LLaMa 3.22025.05 | 47.9 | 49.7 | 53.2 | 47.7 | 48.1 | 47.6 | 46.6 | 46 | 44.2 | |
| LangRankBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 47.3 | — | — | — | — | — | — | — | — | |
| LangSimBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 47.3 | — | — | — | — | — | — | — | — | |
| ASK CALIBackbone=Phi 42025.05 | 44.9 | 38.1 | 42.3 | 49 | 43.5 | 44.2 | 44.2 | 51.4 | 46.7 | |
| RandomBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 44.9 | — | — | — | — | — | — | — | — | |
| TargetBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 44.7 | — | — | — | — | — | — | — | — | |
| AllBackbone=Llama-3.1-Instruct-8B2026.04 | 43.8 | — | — | — | — | — | — | — | — | |
| Pretrained (ZS)Backbone=Phi-4-Mini-Instruct-3.8B, Evaluation Protocol=Zero-shot2026.04 | 43.5 | — | — | — | — | — | — | — | — | |
| AllBackbone=Phi-4-Mini-Instruct-3.8B2026.04 | 38.8 | — | — | — | — | — | — | — | — |