# Scientific Reasoning on ARCc Source: [Scientific Reasoning on ARCc]() Collection index: [llms.txt]() Data updated: 2026-04-23T11:01:31.654Z Evaluation of model performance on the Scientific Reasoning task using the ARCc dataset, measured by Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Scientific Reasoning | | Dataset (including any stated split) | ARCc | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 32 of 32 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | MixLLM | 53.67 | Not recorded | 68fa76491e209131cea6c23b | | float16 | 53.41 | Not recorded | 68fa76491e209131cea6c23b | | SmoothQuant | 53.33 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 53.24 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 52.7 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 52.47 | Not recorded | 68fa76491e209131cea6c23b | | float16 | 52.3 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 52.3 | Not recorded | 68fa76491e209131cea6c23b | | float16 | 52.24 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 52.13 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 52.1 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 51.96 | Not recorded | 68fa76491e209131cea6c23b | | SmoothQuant | 51.88 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 51.88 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 51.88 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 51.88 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 51.82 | Not recorded | 68fa76491e209131cea6c23b | | SmoothQuant | 51.74 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 51.71 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 51.39 | Not recorded | 68fa76491e209131cea6c23b | | QServe | 51.28 | Not recorded | 68fa76491e209131cea6c23b | | QServe | 51.19 | Not recorded | 68fa76491e209131cea6c23b | | float16 | 51.02 | Not recorded | 68fa76491e209131cea6c23b | | MixLLM | 50.94 | Not recorded | 68fa76491e209131cea6c23b | | QServe | 50.54 | Not recorded | 68fa76491e209131cea6c23b | | SmoothQuant | 50 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 50 | Not recorded | 68fa76491e209131cea6c23b | | QServe | 49.15 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 48.63 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 46.33 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 45.99 | Not recorded | 68fa76491e209131cea6c23b | | QuaRot | 43 | Not recorded | 68fa76491e209131cea6c23b | ### MixLLM Accuracy: 53.67. Result ID: 69e9fbee47f34ce924d20bf5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W4.4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### float16 Accuracy: 53.41. Result ID: 69e9fbee47f34ce924d20bdd. [Reporting paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"float16","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### SmoothQuant Accuracy: 53.33. Result ID: 69e9fbee47f34ce924d20be1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 53.24. Result ID: 69e9fbee47f34ce924d20bf9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 52.7. Result ID: 69e9fbee47f34ce924d20bf4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"W4.4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 52.47. Result ID: 69e9fbee47f34ce924d20bf7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W4.4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### float16 Accuracy: 52.3. Result ID: 69e9fbee47f34ce924d20bdf. [Reporting paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"float16","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 52.3. Result ID: 69e9fbee47f34ce924d20bea. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### float16 Accuracy: 52.24. Result ID: 69e9fbee47f34ce924d20bdc. [Reporting paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"float16","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 52.13. Result ID: 69e9fbee47f34ce924d20bfb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 52.1. Result ID: 69e9fbee47f34ce924d20bf8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 51.96. Result ID: 69e9fbee47f34ce924d20bf6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W4.4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### SmoothQuant Accuracy: 51.88. Result ID: 69e9fbee47f34ce924d20be3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 51.88. Result ID: 69e9fbee47f34ce924d20beb. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 51.88. Result ID: 69e9fbee47f34ce924d20bf2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 51.88. Result ID: 69e9fbee47f34ce924d20bf3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 51.82. Result ID: 69e9fbee47f34ce924d20bf0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### SmoothQuant Accuracy: 51.74. Result ID: 69e9fbee47f34ce924d20be0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 51.71. Result ID: 69e9fbee47f34ce924d20bf1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 51.39. Result ID: 69e9fbee47f34ce924d20be8. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Average","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QServe Accuracy: 51.28. Result ID: 69e9fbee47f34ce924d20bed. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QServe Accuracy: 51.19. Result ID: 69e9fbee47f34ce924d20bef. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### float16 Accuracy: 51.02. Result ID: 69e9fbee47f34ce924d20bde. [Reporting paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"float16","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### MixLLM Accuracy: 50.94. Result ID: 69e9fbee47f34ce924d20bfa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QServe Accuracy: 50.54. Result ID: 69e9fbee47f34ce924d20bec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Average","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### SmoothQuant Accuracy: 50. Result ID: 69e9fbee47f34ce924d20be2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W8A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 50. Result ID: 69e9fbee47f34ce924d20be9. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QServe Accuracy: 49.15. Result ID: 69e9fbee47f34ce924d20bee. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W4A8","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 48.63. Result ID: 69e9fbee47f34ce924d20be7. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Mistral-7B-v0.3","Precision":"W4A4","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 46.33. Result ID: 69e9fbee47f34ce924d20be6. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Qwen2.5-7B","Precision":"W4A4","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 45.99. Result ID: 69e9fbee47f34ce924d20be4. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Average","Precision":"W4A4","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. ### QuaRot Accuracy: 43. Result ID: 69e9fbee47f34ce924d20be5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-3.1-8B","Precision":"W4A4","Evaluation Protocol (Shots)":"0-shot"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.