# Reasoning on ARC-C 25-shot Source: [Reasoning on ARC-C 25-shot]() Collection index: [llms.txt]() Data updated: 2026-03-04T12:24:40.151Z Evaluation of model performance on the 25-shot setting of the ARC-C dataset for general reasoning tasks, measured by Score. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Reasoning | | Dataset (including any stated split) | ARC-C 25-shot | | Metric | Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 28 of 28 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | Score | Result date | Report paper | | --- | --- | --- | --- | | UltraMix-190k | 62.43 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-187k | 62.26 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-170k | 62.12 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 60.32 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 59.22 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-190k | 58.7 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-187k | 58.34 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 57.93 | Not recorded | 691aabe56750be83fe5ce08b | | UltraMix-170k | 57.63 | Not recorded | 691aabe56750be83fe5ce08b | | UM-190k | 57.57 | Not recorded | 691aabe56750be83fe5ce08b | | UM-187k | 57.3 | Not recorded | 691aabe56750be83fe5ce08b | | UM-170k | 57.17 | Not recorded | 691aabe56750be83fe5ce08b | | UltraFB | 56.48 | Not recorded | 691aabe56750be83fe5ce08b | | TuluDPO | 56.23 | Not recorded | 691aabe56750be83fe5ce08b | | ORPO | 56.06 | Not recorded | 691aabe56750be83fe5ce08b | | HelpSteer | 55.55 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 55.29 | Not recorded | 691aabe56750be83fe5ce08b | | SFT | 54.95 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-190k) | 53.92 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-187k) | 53.67 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (TuluDPO) | 53.16 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UM-170k) | 53.16 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (HelpSteer) | 52.56 | Not recorded | 691aabe56750be83fe5ce08b | | CodePref | 52.47 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (UltraFB) | 52.3 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (ORPO) | 52.22 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (SFT) | 52.13 | Not recorded | 691aabe56750be83fe5ce08b | | Instella-3B-SFT (CodePref) | 50.77 | Not recorded | 691aabe56750be83fe5ce08b | ### UltraMix-190k Score: 62.43. Result ID: 69a824736dbb7075b94e3f42. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-190k"}. This record does not assert independent reproduction. ### UltraMix-187k Score: 62.26. Result ID: 69a824736dbb7075b94e3f41. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-187k"}. This record does not assert independent reproduction. ### UltraMix-170k Score: 62.12. Result ID: 69a824736dbb7075b94e3f40. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"UM-170k"}. This record does not assert independent reproduction. ### TuluDPO Score: 60.32. Result ID: 69a824736dbb7075b94e3f3f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"TuluDPO"}. This record does not assert independent reproduction. ### SFT Score: 59.22. Result ID: 69a824736dbb7075b94e3f3e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen-2.5-7B-TuluSFT","Training Mixture/Protocol":"SFT"}. This record does not assert independent reproduction. ### UltraMix-190k Score: 58.7. Result ID: 69a824736dbb7075b94e3f3d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-190k"}. This record does not assert independent reproduction. ### UltraMix-187k Score: 58.34. Result ID: 69a824736dbb7075b94e3f3c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-187k"}. This record does not assert independent reproduction. ### TuluDPO Score: 57.93. Result ID: 69a824736dbb7075b94e3f3a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"TuluDPO"}. This record does not assert independent reproduction. ### UltraMix-170k Score: 57.63. Result ID: 69a824736dbb7075b94e3f3b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"UM-170k"}. This record does not assert independent reproduction. ### UM-190k Score: 57.57. Result ID: 69a824736dbb7075b94e4107. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-190k"}. This record does not assert independent reproduction. ### UM-187k Score: 57.3. Result ID: 69a824736dbb7075b94e4106. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-187k"}. This record does not assert independent reproduction. ### UM-170k Score: 57.17. Result ID: 69a824736dbb7075b94e4105. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UM-170k"}. This record does not assert independent reproduction. ### UltraFB Score: 56.48. Result ID: 69a824736dbb7075b94e4102. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"UltraFB"}. This record does not assert independent reproduction. ### TuluDPO Score: 56.23. Result ID: 69a824736dbb7075b94e4100. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"TuluDPO"}. This record does not assert independent reproduction. ### ORPO Score: 56.06. Result ID: 69a824736dbb7075b94e4101. [Reporting paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"ORPO"}. This record does not assert independent reproduction. ### HelpSteer Score: 55.55. Result ID: 69a824736dbb7075b94e4103. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"HelpSteer"}. This record does not assert independent reproduction. ### SFT Score: 55.29. Result ID: 69a824736dbb7075b94e40ff. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"SFT"}. This record does not assert independent reproduction. ### SFT Score: 54.95. Result ID: 69a824736dbb7075b94e3f39. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Llama-3.1-8B-TuluSFT","Training Mixture/Protocol":"SFT"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-190k) Score: 53.92. Result ID: 69a824736dbb7075b94e3e1d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-190k","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-187k) Score: 53.67. Result ID: 69a824736dbb7075b94e3e1c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-187k","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (TuluDPO) Score: 53.16. Result ID: 69a824736dbb7075b94e3e16. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"TuluDPO","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UM-170k) Score: 53.16. Result ID: 69a824736dbb7075b94e3e1b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"DPO Training Dataset":"UM-170k","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (HelpSteer) Score: 52.56. Result ID: 69a824736dbb7075b94e3e19. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"HelpSteer","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### CodePref Score: 52.47. Result ID: 69a824736dbb7075b94e4104. [Reporting paper]() Recorded evaluation configuration: {"Base Model":"OLMo-2-7B-SFT","DPO Training Dataset":"CodePref"}. This record does not assert independent reproduction. ### Instella-3B-SFT (UltraFB) Score: 52.3. Result ID: 69a824736dbb7075b94e3e18. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"UltraFB","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (ORPO) Score: 52.22. Result ID: 69a824736dbb7075b94e3e17. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"ORPO","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (SFT) Score: 52.13. Result ID: 69a824736dbb7075b94e3e15. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"None (SFT Baseline)","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. ### Instella-3B-SFT (CodePref) Score: 50.77. Result ID: 69a824736dbb7075b94e3e1a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"DPO Training Dataset":"CodePref","Evaluation Protocol":"25-shot"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.