# Open-ended MCQA on MMLU (16 subjects filtered) Source: [Open-ended MCQA on MMLU (16 subjects filtered)]() Collection index: [llms.txt]() Data updated: 2026-05-14T07:15:12.825Z Evaluation of model performance on the MMLU benchmark, specifically using the subset filtered to 16 subjects, evaluated using Open-ended Multiple Choice Question Answering methodology. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Open-ended MCQA | | Dataset (including any stated split) | MMLU 16 subjects filtered | | Metric | ASR@30 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 20 of 20 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [ASR@30](): Higher is better; unit: PERCENT. - [Standard Deviation (std)](): Lower is better; unit: SCALAR. - [SCE](): Higher is better; unit: PERCENT. - [SEE (Error Estimate)](): Lower is better; unit: PERCENT. ## Recorded results | Method | ASR@30 | Result date | Report paper | | --- | --- | --- | --- | | REALISTA | 97.11 | Not recorded | 6a05571ddbc836b87d828d2f | | REALISTA | 93.6 | Not recorded | 6a05571ddbc836b87d828d2f | | ICD | 90.77 | Not recorded | 6a05571ddbc836b87d828d2f | | ICD | 87.32 | Not recorded | 6a05571ddbc836b87d828d2f | | LARGO | 84.71 | Not recorded | 6a05571ddbc836b87d828d2f | | SECA | 82.97 | Not recorded | 6a05571ddbc836b87d828d2f | | SECA | 79.61 | Not recorded | 6a05571ddbc836b87d828d2f | | LARGO | 57.92 | Not recorded | 6a05571ddbc836b87d828d2f | | Raw prompting | 54.4 | Not recorded | 6a05571ddbc836b87d828d2f | | Raw prompting | 45.48 | Not recorded | 6a05571ddbc836b87d828d2f | | REALISTA | 41.61 | Not recorded | 6a05571ddbc836b87d828d2f | | SECA | 32.47 | Not recorded | 6a05571ddbc836b87d828d2f | | SECA | 27.51 | Not recorded | 6a05571ddbc836b87d828d2f | | REALISTA | 27.24 | Not recorded | 6a05571ddbc836b87d828d2f | | LARGO | 23.89 | Not recorded | 6a05571ddbc836b87d828d2f | | ICD | 13.56 | Not recorded | 6a05571ddbc836b87d828d2f | | ICD | 11.5 | Not recorded | 6a05571ddbc836b87d828d2f | | LARGO | 8.95 | Not recorded | 6a05571ddbc836b87d828d2f | | Raw prompting | 6.4 | Not recorded | 6a05571ddbc836b87d828d2f | | Raw prompting | 1.62 | Not recorded | 6a05571ddbc836b87d828d2f | ### REALISTA ASR@30: 97.11. Result ID: 6a05767847f34ce924ec0334. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-3B"}. This record does not assert independent reproduction. ### REALISTA ASR@30: 93.6. Result ID: 6a05767847f34ce924ec0339. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-8B"}. This record does not assert independent reproduction. ### ICD ASR@30: 90.77. Result ID: 6a05767847f34ce924ec0333. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-3B"}. This record does not assert independent reproduction. ### ICD ASR@30: 87.32. Result ID: 6a05767847f34ce924ec0338. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-8B"}. This record does not assert independent reproduction. ### LARGO ASR@30: 84.71. Result ID: 6a05767847f34ce924ec0332. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-3B"}. This record does not assert independent reproduction. ### SECA ASR@30: 82.97. Result ID: 6a05767847f34ce924ec0336. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-8B"}. This record does not assert independent reproduction. ### SECA ASR@30: 79.61. Result ID: 6a05767847f34ce924ec0331. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-3B"}. This record does not assert independent reproduction. ### LARGO ASR@30: 57.92. Result ID: 6a05767847f34ce924ec0337. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Llama-3-8B"}. This record does not assert independent reproduction. ### Raw prompting ASR@30: 54.4. Result ID: 6a05767847f34ce924ec0335. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target LLM":"Llama-3-8B"}. This record does not assert independent reproduction. ### Raw prompting ASR@30: 45.48. Result ID: 6a05767847f34ce924ec0330. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target LLM":"Llama-3-3B"}. This record does not assert independent reproduction. ### REALISTA ASR@30: 41.61. Result ID: 6a05767847f34ce924ec033e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-7B"}. This record does not assert independent reproduction. ### SECA ASR@30: 32.47. Result ID: 6a05767847f34ce924ec033b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-7B"}. This record does not assert independent reproduction. ### SECA ASR@30: 27.51. Result ID: 6a05767847f34ce924ec0340. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-14B"}. This record does not assert independent reproduction. ### REALISTA ASR@30: 27.24. Result ID: 6a05767847f34ce924ec0343. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-14B"}. This record does not assert independent reproduction. ### LARGO ASR@30: 23.89. Result ID: 6a05767847f34ce924ec033c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-7B"}. This record does not assert independent reproduction. ### ICD ASR@30: 13.56. Result ID: 6a05767847f34ce924ec0342. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-14B"}. This record does not assert independent reproduction. ### ICD ASR@30: 11.5. Result ID: 6a05767847f34ce924ec033d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-7B"}. This record does not assert independent reproduction. ### LARGO ASR@30: 8.95. Result ID: 6a05767847f34ce924ec0341. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-14B"}. This record does not assert independent reproduction. ### Raw prompting ASR@30: 6.4. Result ID: 6a05767847f34ce924ec033a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-7B"}. This record does not assert independent reproduction. ### Raw prompting ASR@30: 1.62. Result ID: 6a05767847f34ce924ec033f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target LLM":"Qwen-2.5-14B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.