Abstract Reasoning on ConceptARC
97.29AccuracyQwen3 + DIARC
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3 + DIARC#Params=3.6B2026.06 | 97.29 | |
| Qwen3 + SFT only#Params=3.6B2026.06 | 96.46 | |
| Human (pass@1)#Params=-2026.06 | 73 | |
| o4-mini#Params=N/A2026.06 | 70.8 | |
| Gemini 2.5 Pro#Params=N/A2026.06 | 66 | |
| Minitron + DIARC#Params=6.9B2026.06 | 62.71 | |
| Minitron + SFT only#Params=6.9B2026.06 | 60.42 | |
| Claude Sonnet 4#Params=N/A2026.06 | 60.2 | |
| Llama-3.2 + DIARC#Params=2.6B2026.06 | 59.17 | |
| Llama-3.2 + SFT only#Params=2.6B2026.06 | 58.13 | |
| Qwen-2.5#Params=7B2026.06 | 26 | |
| GPT-4#Params=N/A2026.06 | 24 | |
| o4-mini (visual)#Params=N/A2026.06 | 8.1 | |
| Minitron#Params=8B2026.06 | 5.83 | |
| Qwen3#Params=4B2026.06 | 1.25 | |
| Llama-3.2#Params=3B2026.06 | 1.04 |