Commonsense Reasoning on ARC Challenge
97.35AccuracyNemotron-3-Ultra 550B-A55B-Base
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Nemotron-3-Ultra 550B-A55B-Baseshots=25-shot2026.06 | 97.35 | — | — | — | |
| Mistral-Large-3 675B-Base-2512shots=25-shot2026.06 | 97.27 | — | — | — | |
| GLM-4.5 Baseshots=25-shot2026.06 | 96.59 | — | — | — | |
| Kimi-K2 Baseshots=25-shot2026.06 | 95.82 | — | — | — | |
| DeepSeek-V3.2 Exp-Baseshots=25-shot2026.06 | 95.22 | — | — | — | |
| Self-Debias Offline2026.04 | 93.8 | — | — | — | |
| p*LLM Family=Qwen 3 (1.7B and 14B)2026.04 | 93.8 | — | — | — | |
| SGEModel=GPT-4, Tool=Code Interpreter2024.05 | 93.28 | — | — | — | |
| Self-Debias Offline + Self-Correction2026.04 | 93.2 | — | — | — | |
| Self-Debias Iter22026.04 | 93.1 | — | — | — | |
| Self-Debias Iter2 + Self-Correction2026.04 | 93 | — | — | — | |
| Self-Debias SFT2026.04 | 92.9 | — | — | — | |
| Self-Debias Iter1 + Self-Correction2026.04 | 92.8 | — | — | — | |
| Self-Debias Iter12026.04 | 92.5 | — | — | — | |
| Self-Debias SFT + Self-Correction2026.04 | 92.1 | — | — | — | |
| Decomp PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 91.76 | — | — | — | |
| Dual KADLLM Family=Qwen 3 (1.7B and 14B), lambda=0.42026.04 | 91.5 | — | — | — | |
| Dual KADLLM Family=Qwen 3 (1.7B and 14B), lambda=0.32026.04 | 91.4 | — | — | — | |
| Refine PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 91.09 | — | — | — | |
| CoT PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 90.8 | — | — | — | |
| confidence betLLM Family=Qwen 3 (1.7B and 14B), lambda=0.22026.04 | 90.5 | — | — | — | |
| NudgingLLM Family=Qwen 3 (1.7B and 14B), lambda=0.42026.04 | 89.6 | — | — | — | |
| confidence betLLM Family=Qwen 3 (1.7B and 14B), lambda=0.12026.04 | 89.5 | — | — | — | |
| Qwen2.5-7B-Instruct2026.04 | 88.9 | — | — | — | |
| Imp. rewardLLM Family=Qwen 3 (1.7B and 14B)2026.04 | 88.9 | — | — | — | |
| NudgingLLM Family=Qwen 3 (1.7B and 14B), lambda=0.32026.04 | 88.8 | — | — | — | |
| PaLMPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 88.7 | — | — | — | |
| Self-consistencyModel=PaLM-540B2022.03 | 88.7 | — | — | — | |
| Self-AgreementBackbone=GPT-3.5-turbo, Scenario=Question type unknown, answer format known2023.11 | 87.8 | — | — | — | |
| Self-AgreementBackbone=GPT-3.5-turbo, Scenario=Question type and answer format unknown2023.11 | 87.7 | — | — | — | |
| IO PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 87.52 | — | — | — | |
| Self-consistencyModel=GPT-3 (Code-davinci-002)2022.03 | 87.5 | — | — | — | |
| confidence betLLM Family=Qwen 3 (1.7B and 14B), lambda=02026.04 | 87.3 | — | — | — | |
| GPT-3.5-turboPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 87 | — | — | — | |
| Self-AgreementModel=GPT-3.5-turbo, Scenario=Scenario 32023.11 | 86.8 | — | — | — | |
| pLLM Family=Qwen 3 (1.7B and 14B)2026.04 | 86.6 | — | — | — | |
| Mixed-Few-Shot CoTBackbone=GPT-3.5-turbo, Scenario=Question type unknown, answer format known2023.11 | 86 | — | — | — | |
| GPT-3.5-turboPrompting Strategy=Few-Shot CoT, Scenario=Scenario 32023.11 | 85.4 | — | — | — | |
| PaLMPrompting Strategy=Few-Shot CoT, Scenario=Scenario 32023.11 | 85.2 | — | — | — | |
| CoT-promptingModel=PaLM-540B2022.03 | 85.2 | — | — | — | |
| Qwen2.5-7B-Instruct + Self-Correction2026.04 | 85 | — | — | — | |
| DeepSeek-R1-Distill-Qwen-7B2026.04 | 83.8 | — | — | — | |
| Qwen1.5-8B2026.04 | 83.7 | — | — | — | |
| CoT-promptingModel=GPT-3 (Code-davinci-002)2022.03 | 83.6 | — | — | — | |
| Self-ConsistencyBackbone=GPT-3.5-turbo, Scenario=Question type unknown, answer format known2023.11 | 83.1 | — | — | — | |
| q*LLM Family=Qwen 3 (1.7B and 14B)2026.04 | 82.9 | — | — | — | |
| HiRAModel=Llama-3-8B, Params (%)=0.70022026.04 | 82.9 | — | — | — | |
| p*LLM Family=OLMo 2 (1B and 13B)2026.04 | 82.5 | — | — | — | |
| DeepSeek-R1-Distill-Qwen-7B + Self-Correction2026.04 | 81.9 | — | — | — | |
| Qwen1.5-8B + Self-Correction2026.04 | 81.7 | — | — | — | |
| Zero-Shot CoTBackbone=GPT-3.5-turbo, Scenario=Question type and answer format unknown2023.11 | 81.5 | — | — | — | |
| BaLoRAModel=Llama-3-8B, Params (%)=0.71452026.04 | 80.9 | — | — | — | |
| DoRAModel=Llama-3-8B, Params (%)=0.70022026.04 | 80.4 | — | — | — | |
| ChatGPTModel=ChatGPT2026.04 | 79.9 | — | — | — | |
| MoRAModel=Llama-3-8B, Params (%)=0.69972026.04 | 79.61 | — | — | — | |
| Llama-3.1-8B-Instruct2026.04 | 78.6 | — | — | — | |
| Fine-tuned SOTASetting=Fine-tuned2020.05 | 78.5 | — | — | — | |
| Multi-Agents (Debate)Backbone=GPT-3.5-turbo, Scenario=Question type unknown, answer format known2023.11 | 78.4 | — | — | — | |
| UnifiedQA-FTScenario=Scenario 32023.11 | 75 | — | — | — | |
| Previous SoTA2022.03 | 75 | — | — | — | |
| GPT-3.5-turboPrompting Strategy=USC, Scenario=Scenario 32023.11 | 73.9 | — | — | — | |
| HiRAModel=Llama-2-7B, Params (%)=0.82562026.04 | 73.81 | — | — | — | |
| BaLoRAModel=Llama-2-7B, Params (%)=0.84272026.04 | 72.1 | — | — | — | |
| Llama-3.1-8B-Instruct + Self-Correction2026.04 | 71.9 | — | — | — | |
| MoRAModel=Llama-2-7B, Params (%)=0.82412026.04 | 71.42 | — | — | — | |
| LoRAModel=Llama-3-8B, Params (%)=0.70022026.04 | 71.2 | — | — | — | |
| Self-AgreementBackbone=Llama-2-13B-Chat, Scenario=Question type unknown, answer format known2023.11 | 70.6 | — | — | — | |
| Bayesian-LoRA (S=4)S (Samples)=4, Evaluation Protocol=End-to-End, rind=9, cind=92026.01 | 68.9 | 9.2 | 0.77 | — | |
| BLoB (N=10)N (Ensemble Size)=10, rind=9, cind=92026.01 | 68.81 | 9.59 | 0.78 | — | |
| Self-AgreementBackbone=Llama-2-13B-Chat, Scenario=Question type and answer format unknown2023.11 | 68.5 | — | — | — | |
| DoRAModel=Llama-2-7B, Params (%)=0.82562026.04 | 68.2 | — | — | — | |
| BBBrind=9, cind=92026.01 | 68.13 | 26.23 | 2.23 | — | |
| Self-ConsistencyBackbone=Llama-2-13B-Chat, Scenario=Question type unknown, answer format known2023.11 | 68.1 | — | — | — | |
| Mixed-Few-Shot CoTBackbone=Llama-2-13B-Chat, Scenario=Question type unknown, answer format known2023.11 | 68 | — | — | — | |
| LA (post-hoc)Evaluation Protocol=post-hoc, rind=9, cind=92026.01 | 66.9 | 7.5 | 0.86 | — | |
| Zero-Shot CoTBackbone=Llama-2-13B-Chat, Scenario=Question type and answer format unknown2023.11 | 66.5 | — | — | — | |
| LLLA (post-hoc)Evaluation Protocol=post-hoc, rind=9, cind=92026.01 | 66.1 | 5.6 | 0.94 | — | |
| MAPrind=9, cind=92026.01 | 64.9 | 26.1 | 1.64 | — | |
| Dropoutrind=9, cind=92026.01 | 64.9 | 25.6 | 1.55 | — | |
| Ckpt Ensrind=9, cind=92026.01 | 64.9 | 26.1 | 1.64 | — | |
| Temprind=9, cind=92026.01 | 64.9 | 4.6 | 0.9 | — | |
| LoRAModel=Llama-2-7B, Params (%)=0.82562026.04 | 64.7 | — | — | — | |
| Imp. rewardLLM Family=OLMo 2 (1B and 13B)2026.04 | 63.3 | — | — | — | |
| confidence betLLM Family=OLMo 2 (1B and 13B), lambda=0.22026.04 | 62.6 | — | — | — | |
| Dual KADLLM Family=OLMo 2 (1B and 13B), lambda=0.42026.04 | 61.9 | — | — | — | |
| Dual KADLLM Family=OLMo 2 (1B and 13B), lambda=0.32026.04 | 61.6 | — | — | — | |
| LaMDAPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 59.8 | — | — | — | |
| Self-consistencyModel=LaMDA-137B2022.03 | 59.8 | — | — | — | |
| confidence betLLM Family=OLMo 2 (1B and 13B), lambda=0.12026.04 | 59.5 | — | — | — | |
| LLaMAParameters=33B, Zero-shot=true, Likelihood Normalization=Per-character2023.02 | 57.8 | — | — | — | |
| NudgingLLM Family=OLMo 2 (1B and 13B), lambda=0.42026.04 | 57.8 | — | — | — | |
| confidence betLLM Family=OLMo 2 (1B and 13B), lambda=02026.04 | 56.6 | — | — | — | |
| Multi-Agents (Debate)Backbone=Llama-2-13B-Chat, Scenario=Question type unknown, answer format known2023.11 | 56.5 | — | — | — | |
| LLaMAParameters=65B, Zero-shot=true, Likelihood Normalization=Per-character2023.02 | 56 | — | — | — | |
| LaMDAPrompting Strategy=Few-Shot CoT, Scenario=Scenario 32023.11 | 55.1 | — | — | — | |
| CoT-promptingModel=LaMDA-137B2022.03 | 55.1 | — | — | — | |
| Self-consistencyModel=GPT-3 (Code-davinci-001)2022.03 | 53.7 | — | — | — | |
| NudgingLLM Family=OLMo 2 (1B and 13B), lambda=0.32026.04 | 53.7 | — | — | — | |
| KEELArchitecture=512 Layers / 3B Params, Peak Learning Rate=4.5 x 10^-3, Pre-training Tokens=1T, Evaluation Protocol (Shots)=25-shot2026.01 | 53.6 | — | — | — | |
| GPT-3Setting=One-Shot2020.05 | 53.2 | — | — | — |