# Knowledge on MMLU-Pro (5-shot) Source: [Knowledge on MMLU-Pro (5-shot)]() Collection index: [llms.txt]() Data updated: 2026-03-04T12:24:40.073Z Evaluation of model performance on the Knowledge subject category within the MMLU-Pro benchmark using a 5-shot prompting setup, measured by Score. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Knowledge | | Dataset (including any stated split) | MMLU-Pro 5-shot | | Metric | Knowledge Score (5-shot) | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 37 of 37 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Knowledge Score (5-shot)](): Higher is better; unit: PERCENT. ## Recorded results | Method | Knowledge Score (5-shot) | Result date | Report paper | | --- | --- | --- | --- | | UltraMix-190k | 44.65 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-187k | 44.5 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 43.48 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 43.32 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-170k | 43.16 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-190k | 30.96 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-187k | 30.24 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 28.98 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-170k | 28.5 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 28.08 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-190k) | 27.86 | Not recorded | 691aabe56750be83fe5ce08b | | UM-190k | 27.84 | Not recorded | 691aabe56750be83fe5ce08b | | UM-187k | 27.69 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-187k) | 27.17 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 27.1 | Not recorded | 691aabe56750be83fe5ce08b | | UM-170k | 27.1 | Not recorded | 691aabe56750be83fe5ce08b | | UltraFB | 26.75 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-170k) | 26.44 | Not recorded | 691aabe56750be83fe5ce08b | | ORPO | 26.39 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (TuluDPO) | 26.06 | Not recorded | 691aabe56750be83fe5ce08b | | HelpSteer | 26.02 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (ORPO) | 25.95 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 25.94 | Not recorded | 691aabe56750be83fe5ce08b | | CodePref | 25.78 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (SFT) | 25.44 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (HelpSteer) | 25.39 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (CodePref) | 25.19 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UltraFB) | 25.14 | Not recorded | 691aabe56750be83fe5ce08b | | UM-190k | 14.77 | Not recorded | 691aabe56750be83fe5ce08b | | UM-187k | 14.64 | Not recorded | 691aabe56750be83fe5ce08b | | CodePref | 14.28 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 14.07 | Not recorded | 691aabe56750be83fe5ce08b | | ORPO | 14.01 | Not recorded | 691aabe56750be83fe5ce08b | | UM-170k | 14 | Not recorded | 691aabe56750be83fe5ce08b | | UltraFB | 13.45 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 13.02 | Not recorded | 691aabe56750be83fe5ce08b | | HelpSteer | 12.83 | Not recorded | 691aabe56750be83fe5ce08b | ### UltraMix-190k Knowledge Score (5-shot): 44.65. Result ID: 69a824736dbb7075b94e3f24. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-190k"}. This record does not assert independent reproduction. ### UltraMix-187k Knowledge Score (5-shot): 44.5. Result ID: 69a824736dbb7075b94e3f23. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-187k"}. This record does not assert independent reproduction. ### TuluDPO Knowledge Score (5-shot): 43.48. Result ID: 69a824736dbb7075b94e3f21. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"TuluDPO"}. This record does not assert independent reproduction. ### SFT Knowledge Score (5-shot): 43.32. Result ID: 69a824736dbb7075b94e3f20. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"SFT"}. This record does not assert independent reproduction. ### UltraMix-170k Knowledge Score (5-shot): 43.16. Result ID: 69a824736dbb7075b94e3f22. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-170k"}. This record does not assert independent reproduction. ### UltraMix-190k Knowledge Score (5-shot): 30.96. Result ID: 69a824736dbb7075b94e3f1f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-190k"}. This record does not assert independent reproduction. ### UltraMix-187k Knowledge Score (5-shot): 30.24. Result ID: 69a824736dbb7075b94e3f1e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-187k"}. This record does not assert independent reproduction. ### TuluDPO Knowledge Score (5-shot): 28.98. Result ID: 69a824736dbb7075b94e3f1c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"TuluDPO"}. This record does not assert independent reproduction. ### UltraMix-170k Knowledge Score (5-shot): 28.5. Result ID: 69a824736dbb7075b94e3f1d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-170k"}. This record does not assert independent reproduction. ### SFT Knowledge Score (5-shot): 28.08. Result ID: 69a824736dbb7075b94e3f1b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"SFT"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-190k) Knowledge Score (5-shot): 27.86. Result ID: 69a824736dbb7075b94e3e02. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-190k","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### UM-190k Knowledge Score (5-shot): 27.84. Result ID: 69a824736dbb7075b94e40f5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-190k"}. This record does not assert independent reproduction. ### UM-187k Knowledge Score (5-shot): 27.69. Result ID: 69a824736dbb7075b94e40f4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-187k"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-187k) Knowledge Score (5-shot): 27.17. Result ID: 69a824736dbb7075b94e3e01. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-187k","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### TuluDPO Knowledge Score (5-shot): 27.1. Result ID: 69a824736dbb7075b94e40ee. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"TuluDPO"}. This record does not assert independent reproduction. ### UM-170k Knowledge Score (5-shot): 27.1. Result ID: 69a824736dbb7075b94e40f3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-170k"}. This record does not assert independent reproduction. ### UltraFB Knowledge Score (5-shot): 26.75. Result ID: 69a824736dbb7075b94e40f0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UltraFB"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-170k) Knowledge Score (5-shot): 26.44. Result ID: 69a824736dbb7075b94e3e00. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-170k","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### ORPO Knowledge Score (5-shot): 26.39. Result ID: 69a824736dbb7075b94e40ef. [Reporting paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"ORPO"}. This record does not assert independent reproduction. ### Instella-3B-SFT (TuluDPO) Knowledge Score (5-shot): 26.06. Result ID: 69a824736dbb7075b94e3dfb. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"TuluDPO","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### HelpSteer Knowledge Score (5-shot): 26.02. Result ID: 69a824736dbb7075b94e40f1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"HelpSteer"}. This record does not assert independent reproduction. ### Instella-3B-SFT (ORPO) Knowledge Score (5-shot): 25.95. Result ID: 69a824736dbb7075b94e3dfc. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"ORPO","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### SFT Knowledge Score (5-shot): 25.94. Result ID: 69a824736dbb7075b94e40ed. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"SFT"}. This record does not assert independent reproduction. ### CodePref Knowledge Score (5-shot): 25.78. Result ID: 69a824736dbb7075b94e40f2. [Reporting paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"CodePref"}. This record does not assert independent reproduction. ### Instella-3B-SFT (SFT) Knowledge Score (5-shot): 25.44. Result ID: 69a824736dbb7075b94e3dfa. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"None (SFT Baseline)","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (HelpSteer) Knowledge Score (5-shot): 25.39. Result ID: 69a824736dbb7075b94e3dfe. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"HelpSteer","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (CodePref) Knowledge Score (5-shot): 25.19. Result ID: 69a824736dbb7075b94e3dff. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"CodePref","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UltraFB) Knowledge Score (5-shot): 25.14. Result ID: 69a824736dbb7075b94e3dfd. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"UltraFB","Evaluation Protocol":"5-shot"}. This record does not assert independent reproduction. ### UM-190k Knowledge Score (5-shot): 14.77. Result ID: 69a824736dbb7075b94e3fcc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"UM-190k"}. This record does not assert independent reproduction. ### UM-187k Knowledge Score (5-shot): 14.64. Result ID: 69a824736dbb7075b94e3fcb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"UM-187k"}. This record does not assert independent reproduction. ### CodePref Knowledge Score (5-shot): 14.28. Result ID: 69a824736dbb7075b94e3fc9. [Reporting paper]() Recorded evaluation configuration: {"Training Dataset":"CodePref"}. This record does not assert independent reproduction. ### TuluDPO Knowledge Score (5-shot): 14.07. Result ID: 69a824736dbb7075b94e3fc5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"TuluDPO"}. This record does not assert independent reproduction. ### ORPO Knowledge Score (5-shot): 14.01. Result ID: 69a824736dbb7075b94e3fc6. [Reporting paper]() Recorded evaluation configuration: {"Training Dataset":"ORPO"}. This record does not assert independent reproduction. ### UM-170k Knowledge Score (5-shot): 14. Result ID: 69a824736dbb7075b94e3fca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"UM-170k"}. This record does not assert independent reproduction. ### UltraFB Knowledge Score (5-shot): 13.45. Result ID: 69a824736dbb7075b94e3fc7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"UltraFB"}. This record does not assert independent reproduction. ### SFT Knowledge Score (5-shot): 13.02. Result ID: 69a824736dbb7075b94e3fc4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"SFT"}. This record does not assert independent reproduction. ### HelpSteer Knowledge Score (5-shot): 12.83. Result ID: 69a824736dbb7075b94e3fc8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Training Dataset":"HelpSteer"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.