Reasoning on ARC Challenge
96.7AccuracyGPT-4o
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-4o2024.10 | 96.7 | — | |
| GPT-4o mini2024.10 | 96.4 | — | |
| GPT-4Zero-shot=true2023.11 | 93.26 | — | |
| ARIA2024.10 | 91 | — | |
| GRPOBackbone=Qwen3-4B2025.10 | 90.1 | — | |
| ASPO-Math-8BBackbone=Qwen3-8B2025.10 | 88.6 | — | |
| ASPO-Math-4BBackbone=Qwen3-4B2025.10 | 87.6 | — | |
| ChatGPTZero-shot=true2023.11 | 84.73 | — | |
| Llama3.2-11B2024.10 | 83.4 | — | |
| Orca-2-13BZero-shot=true, System Message=empty2023.11 | 83.36 | — | |
| Orca-2-13BZero-shot=true, System Message=cautious2023.11 | 79.95 | — | |
| Orca-2-7BZero-shot=true, System Message=empty2023.11 | 78.41 | — | |
| Orca-2-7BZero-shot=true, System Message=cautious2023.11 | 74.83 | — | |
| Orca-1-13BZero-shot=true2023.11 | 74.74 | — | |
| Qwen3-4BBackbone=Qwen3-4B2025.10 | 74.7 | — | |
| WizardLM-70BZero-shot=true2023.11 | 71.93 | — | |
| Mixtral-8x22B2024.07 | 70.7 | — | |
| Qwen1.5-110B2024.07 | 69.6 | — | |
| Qwen2-72B2024.07 | 68.9 | — | |
| Llama-3-70B2024.07 | 68.8 | — | |
| LLAMA-2-Chat-70BZero-shot=true2023.11 | 67.66 | — | |
| Qwen1.5-72B2024.07 | 65.9 | — | |
| GRPOBackbone=Qwen3-8B2025.10 | 62.4 | — | |
| LLAMA-2-Chat-13BZero-shot=true2023.11 | 61.18 | — | |
| Phi-2# Non-Emb Params=2.5B2024.07 | 61.1 | — | |
| Qwen3.5-9BArchitecture Type=Hybrid2026.04 | 54.77 | — | |
| LLAMA-3-8BModel=LLAMA-3-8B, Bits=FP16, Zero-shot=true2026.04 | 53.67 | — | |
| Qwen3-8BBackbone=Qwen3-8B2025.10 | 52.9 | — | |
| Qwen3.5-4BArchitecture Type=Hybrid2026.04 | 51.62 | — | |
| Qwen3-4BArchitecture Type=O(n^2)2026.04 | 50.68 | — | |
| WizardLM-13BZero-shot=true2023.11 | 50.43 | — | |
| Learnable ResFormerEvaluation Protocol=Zero-shot2024.10 | 50.3 | — | |
| Learnable ResFormer plusEvaluation Protocol=Zero-shot2024.10 | 49.7 | — | |
| TransformerEvaluation Protocol=Zero-shot2024.10 | 49.6 | — | |
| Identity ResFormerEvaluation Protocol=Zero-shot2024.10 | 49 | — | |
| Zamba2-2.7BArchitecture Type=Hybrid2026.04 | 48.98 | — | |
| NeuTRENOEvaluation Protocol=Zero-shot2024.10 | 48.8 | — | |
| DenseFormerEvaluation Protocol=Zero-shot2024.10 | 48.7 | — | |
| LayerBoost-Qwen3-4BArchitecture Type=Hybrid2026.04 | 48.54 | — | |
| Gemma-2B# Non-Emb Params=2.0B2024.07 | 48.5 | — | |
| RWKV7-W3-2.9BArchitecture Type=O(n)2026.04 | 48.21 | — | |
| Qwen2-1.5B# Non-Emb Params=1.2B2024.07 | 43.9 | — | |
| Llama-3.2-3B-Inst.Architecture Type=O(n^2)2026.04 | 43.6 | — | |
| LLAMA-1-7BModel=LLAMA-1-7B, Bits=FP16, Zero-shot=true2026.04 | 41.38 | — | |
| Qwen2.5-1.5BArchitecture Type=O(n^2)2026.04 | 41.38 | — | |
| LLAMA-2-7BModel=LLAMA-2-7B, Bits=FP16, Zero-shot=true2026.04 | 40.53 | — | |
| Qwen3-1.7BArchitecture Type=O(n^2)2026.04 | 39.93 | — | |
| Jet-Nemotron-4BArchitecture Type=Hybrid2026.04 | 39.76 | — | |
| Qwen3.5-2BArchitecture Type=Hybrid2026.04 | 38.31 | — | |
| Qwen1.5-1.8B# Non-Emb Params=1.2B2024.07 | 37.9 | — | |
| Jet-Nemotron-2BArchitecture Type=Hybrid2026.04 | 37.46 | — | |
| NAG_Llama-3.2-3BTarget Setting=Single-Target, Backbone=Llama-3.2-3B2026.04 | 35 | — | |
| NAG_SmolLM3-3BTarget Setting=Single-Target, Backbone=SmolLM3-3B2026.04 | 35 | — | |
| FineWeb-Edu2026.04 | 34.3 | — | |
| Mamba-2.7BArchitecture Type=O(n)2026.04 | 34.22 | — | |
| NAG_Qwen3-1.7BTarget Setting=Single-Target, Backbone=Qwen3-1.7B2026.04 | 34 | — | |
| LBLLMModel=LLAMA-1-7B, Bits=W(1+1)A4, Zero-shot=true2026.04 | 33.53 | — | |
| NAG_Qwen3-1.7BTarget Setting=Multi-Target, Backbone=Qwen3-1.7B2026.04 | 33.4 | — | |
| LBLLMModel=LLAMA-2-7B, Bits=W(1+1)A4, Zero-shot=true2026.04 | 32.51 | — | |
| BETRTarget Setting=Single-Target2026.04 | 32.3 | — | |
| LBLLMModel=LLAMA-3-8B, Bits=W(1+1)A4, Zero-shot=true2026.04 | 32.17 | — | |
| NAG_Llama-3.2-3BTarget Setting=Multi-Target, Backbone=Llama-3.2-3B2026.04 | 32 | — | |
| NAG_SmolLM3-3BTarget Setting=Multi-Target, Backbone=SmolLM3-3B2026.04 | 31.8 | — | |
| Qwen2-0.5B# Non-Emb Params=0.3B2024.07 | 31.5 | — | |
| BETRTarget Setting=Multi-Target2026.04 | 30.3 | — | |
| Random2026.04 | 28.5 | — | |
| Gemma-3-4B-itArchitecture Type=Hybrid2026.04 | 27.55 | — | |
| PDPCSelection Metric=PD, Selection Order=S., Model Size=1.3B2025.02 | 26.6 | — | |
| FRAMESelection Metric=PPL&PD, Model Size=1.3B2025.02 | 26.5 | — | |
| SequentialSelection Metric=Qu.Edu, Selection Order=High2Low, Model Size=1.3B2025.02 | 26.4 | — | |
| SequentialSelection Metric=Qu.Edu, Selection Order=Low2High, Model Size=1.3B2025.02 | 26 | — | |
| Preference CLSelection Metric=Qu.Edu, Selection Order=S.R., Model Size=1.3B2025.02 | 24.9 | — | |
| Preference CLSelection Metric=PD, Selection Order=S.R., Model Size=1.3B2025.02 | 24.9 | — | |
| Preference CLSelection Metric=Qu.Edu, Selection Order=S., Model Size=1.3B2025.02 | 24.8 | — | |
| Preference CLSelection Metric=PPL, Selection Order=S.R., Model Size=1.3B2025.02 | 24.1 | — | |
| RandomModel Size=1.3B2025.02 | 23.6 | — | |
| Preference CLSelection Metric=PPL, Selection Order=S., Model Size=1.3B2025.02 | 22.6 | — | |
| SequentialSelection Metric=PD, Selection Order=High2Low, Model Size=1.3B2025.02 | 21.8 | — | |
| SequentialSelection Metric=PD, Selection Order=Low2High, Model Size=1.3B2025.02 | 21.3 | — | |
| SequentialSelection Metric=PPL, Selection Order=High2Low, Model Size=1.3B2025.02 | 20.6 | — | |
| SequentialSelection Metric=PPL, Selection Order=Low2High, Model Size=1.3B2025.02 | 17.9 | — | |
| BaseBackbone=Qwen2.5-7B-Instruct2025.08 | — | 91.54 | |
| BaseBackbone=Llama3.1-8B-Instruct2025.08 | — | 80.96 | |
| DeepSeek-R1-Distill-1.5BBackbone=DeepSeek-R1-Distill-1.5B, Strategy=Base2026.02 | — | 43.1 | |
| DeepSeek-R1-Distill-7BBackbone=DeepSeek-R1-Distill-7B, Strategy=Base2026.02 | — | 76.1 | |
| Dynamic Sampling (Oracle)Backbone=DeepSeek-R1-Distill-1.5B, Strategy=DS, Runtime=30h2026.02 | — | 46.7 | |
| Dynamic Sampling (Oracle)Backbone=DeepSeek-R1-Distill-7B, Strategy=DS, Runtime=77h2026.02 | — | 78.2 | |
| GPSBackbone=DeepSeek-R1-Distill-1.5B, Strategy=GPS, Runtime=16h2026.02 | — | 46.9 | |
| GPSBackbone=DeepSeek-R1-Distill-7B, Strategy=GPS, Runtime=49h2026.02 | — | 79.7 | |
| GRESOBackbone=DeepSeek-R1-Distill-1.5B, Strategy=GRESO, Runtime=27h2026.02 | — | 46.7 | |
| GRESOBackbone=DeepSeek-R1-Distill-7B, Strategy=GRESO, Runtime=53h2026.02 | — | 77.1 | |
| MoPPSBackbone=DeepSeek-R1-Distill-1.5B, Strategy=MoPPS, Runtime=17h2026.02 | — | 44.6 | |
| MoPPSBackbone=DeepSeek-R1-Distill-7B, Strategy=MoPPS, Runtime=42h2026.02 | — | 76.7 | |
| PCLBackbone=DeepSeek-R1-Distill-1.5B, Strategy=PCL, Runtime=17h2026.02 | — | 46.3 | |
| PCLBackbone=DeepSeek-R1-Distill-7B, Strategy=PCL, Runtime=50h2026.02 | — | 74.4 | |
| PSFTBackbone=Qwen2.5-7B-Instruct2025.08 | — | 92.29 | |
| PSFTBackbone=Llama3.1-8B-Instruct2025.08 | — | 89.11 | |
| PSFTwarm-upBackbone=Qwen2.5-7B-Instruct2025.08 | — | 92.09 | |
| PSFTwarm-upBackbone=Llama3.1-8B-Instruct2025.08 | — | 89.72 | |
| SFTBackbone=Qwen2.5-7B-Instruct2025.08 | — | 92.22 |