# Multilingual Language Understanding on MMMLU Source: [Multilingual Language Understanding on MMMLU]() Collection index: [llms.txt]() Data updated: 2026-03-06T07:48:30.962Z Evaluation of model performance on the Massive Multilingual Language Understanding (MMMLU) benchmark, focusing on specific metrics like CLCall, AEN, and A¬EN. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Multilingual Language Understanding | | Dataset (including any stated split) | MMMLU | | Metric | CLCall | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [CLCall](): Higher is better; unit: SCALAR. - [AEN](): Higher is better; unit: SCALAR. - [A¬EN](): Higher is better; unit: SCALAR. ## Recorded results | Method | CLCall | Result date | Report paper | | --- | --- | --- | --- | | Base | 76.1 | Not recorded | 69aa5f652d603e6de3e41703 | | Base | 73.6 | Not recorded | 69aa5f652d603e6de3e41703 | | Base | 72.2 | Not recorded | 69aa5f652d603e6de3e41703 | | Base | 68.6 | Not recorded | 69aa5f652d603e6de3e41703 | | Base | 60.9 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 13.8 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 13.1 | Not recorded | 69aa5f652d603e6de3e41703 | | DPO | 12.3 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 10.6 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 10.2 | Not recorded | 69aa5f652d603e6de3e41703 | | DPO | 10.1 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 9.4 | Not recorded | 69aa5f652d603e6de3e41703 | | DPO | 6.5 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 6.5 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 5.3 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 4.4 | Not recorded | 69aa5f652d603e6de3e41703 | | SFT | 4.3 | Not recorded | 69aa5f652d603e6de3e41703 | | CALM | 4.2 | Not recorded | 69aa5f652d603e6de3e41703 | | SFT | 3.5 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 3.1 | Not recorded | 69aa5f652d603e6de3e41703 | | CALM | 3 | Not recorded | 69aa5f652d603e6de3e41703 | | DPO | 3 | Not recorded | 69aa5f652d603e6de3e41703 | | CALM | 3 | Not recorded | 69aa5f652d603e6de3e41703 | | DCO | 2.7 | Not recorded | 69aa5f652d603e6de3e41703 | | CALM | 1.4 | Not recorded | 69aa5f652d603e6de3e41703 | | DPO | 1.3 | Not recorded | 69aa5f652d603e6de3e41703 | | SFT | 0.6 | Not recorded | 69aa5f652d603e6de3e41703 | | SFT | 0.6 | Not recorded | 69aa5f652d603e6de3e41703 | | CALM | 0.3 | Not recorded | 69aa5f652d603e6de3e41703 | | SFT | -0.2 | Not recorded | 69aa5f652d603e6de3e41703 | ### Base CLCall: 76.1. Result ID: 69aa85ec6dbb7075b95403fe. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### Base CLCall: 73.6. Result ID: 69aa85ec6dbb7075b95403f8. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### Base CLCall: 72.2. Result ID: 69aa85ec6dbb7075b9540404. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### Base CLCall: 68.6. Result ID: 69aa85ec6dbb7075b95403f2. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### Base CLCall: 60.9. Result ID: 69aa85ec6dbb7075b954040a. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DCO CLCall: 13.8. Result ID: 69aa85ec6dbb7075b954040d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DCO CLCall: 13.1. Result ID: 69aa85ec6dbb7075b95403f5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DPO CLCall: 12.3. Result ID: 69aa85ec6dbb7075b95403f4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DCO CLCall: 10.6. Result ID: 69aa85ec6dbb7075b95403f7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DCO CLCall: 10.2. Result ID: 69aa85ec6dbb7075b95403fb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DPO CLCall: 10.1. Result ID: 69aa85ec6dbb7075b954040c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DCO CLCall: 9.4. Result ID: 69aa85ec6dbb7075b954040f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DPO CLCall: 6.5. Result ID: 69aa85ec6dbb7075b95403fa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DCO CLCall: 6.5. Result ID: 69aa85ec6dbb7075b95403fd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DCO CLCall: 5.3. Result ID: 69aa85ec6dbb7075b9540409. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DCO CLCall: 4.4. Result ID: 69aa85ec6dbb7075b9540401. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### SFT CLCall: 4.3. Result ID: 69aa85ec6dbb7075b954040b. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### CALM CLCall: 4.2. Result ID: 69aa85ec6dbb7075b95403f6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### SFT CLCall: 3.5. Result ID: 69aa85ec6dbb7075b9540405. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### DCO CLCall: 3.1. Result ID: 69aa85ec6dbb7075b9540407. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### CALM CLCall: 3. Result ID: 69aa85ec6dbb7075b95403fc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DPO CLCall: 3. Result ID: 69aa85ec6dbb7075b9540400. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### CALM CLCall: 3. Result ID: 69aa85ec6dbb7075b954040e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama3.1-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DCO CLCall: 2.7. Result ID: 69aa85ec6dbb7075b9540403. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### CALM CLCall: 1.4. Result ID: 69aa85ec6dbb7075b9540408. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### DPO CLCall: 1.3. Result ID: 69aa85ec6dbb7075b9540406. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Aya-Expanse-8B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### SFT CLCall: 0.6. Result ID: 69aa85ec6dbb7075b95403f3. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### SFT CLCall: 0.6. Result ID: 69aa85ec6dbb7075b95403f9. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Gemma3-12B-pt","ground\_truth\_training":"true"}. This record does not assert independent reproduction. ### CALM CLCall: 0.3. Result ID: 69aa85ec6dbb7075b9540402. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"false"}. This record does not assert independent reproduction. ### SFT CLCall: -0.2. Result ID: 69aa85ec6dbb7075b95403ff. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3-14B","ground\_truth\_training":"true"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.