Code Reasoning on CRUXEval (CoT Metrics)
98.8Input-CoT AccuracyGemini-3-Pro-preview
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini-3-Pro-previewModel Size=Closed-APIs2026.03 | 98.8 | 99.1 | |
| Claude-Opus-4.5Model Size=Closed-APIs2026.03 | 98.4 | 98 | |
| Gemini-3-Flash-previewModel Size=Closed-APIs2026.03 | 96.5 | 97.6 | |
| Claude-Sonnet-4.5Model Size=Closed-APIs2026.03 | 96.2 | 96.2 | |
| IQuest-Coder-V1-40B-InstructModel Size=20B+, Variant=Instruct2026.03 | 93.5 | 87 | |
| Kimi-K2-ThinkingModel Size=20B+, Variant=Thinking2026.03 | 92.2 | 86.2 | |
| IQuest-Coder-V1-40B-Loop-InstructModel Size=20B+, Variant=Loop-Instruct2026.03 | 91.1 | 85.5 | |
| IQuest-Coder-V1-40B-ThinkingModel Size=20B+, Variant=Thinking2026.03 | 87.4 | 94 | |
| Qwen3-Coder-480B-A35B-InstructModel Size=20B+, Variant=Instruct2026.03 | 87.1 | 90.4 | |
| Kimi-K2-Instruct-0905Model Size=20B+, Variant=Instruct2026.03 | 86.8 | 89.5 | |
| DeepSeek-v3.2Model Size=20B+2026.03 | 82.1 | 94.2 | |
| IQuest-Coder-V1-14B-ThinkingModel Size=13B+, Variant=Thinking2026.03 | 80.5 | 90.6 | |
| Qwen2.5-Coder-32B-InstructModel Size=20B+, Variant=Instruct2026.03 | 78.8 | 84 | |
| Qwen3-Coder-30B-A3B-InstructModel Size=13B+, Variant=Instruct2026.03 | 76.9 | 80.5 | |
| IQuest-Coder-V1-40B-Loop-ThinkingModel Size=20B+, Variant=Loop-Thinking2026.03 | 76.5 | 75.2 | |
| Qwen2.5-Coder-14B-InstructModel Size=13B+, Variant=Instruct2026.03 | 75.6 | 79.2 | |
| Qwen3-8BSize=8B2026.01 | 73.8 | 76.9 | |
| KAT-Dev-72B-ExpModel Size=20B+2026.03 | 71.4 | 81.1 | |
| GPT-5.1Model Size=Closed-APIs2026.03 | 70.8 | 71.1 | |
| Qwen2.5-Coder-14B-InstructSize=14B2026.01 | 69.5 | 79.5 | |
| Qwen2.5-Coder-7B-InstructModel Size=6B+, Variant=Instruct2026.03 | 66.9 | 66 | |
| Qwen2.5-Coder-7B-InstructSize=7B2026.01 | 65.8 | 65.9 | |
| GLM-4.7Model Size=20B+2026.03 | 65.6 | 81.2 | |
| Seed-Coder-8B-InstructSize=8B2026.01 | 63.3 | 67.1 | |
| Stable-DiffCoder-8B-InstructSize=8B2026.01 | 62.1 | 69 | |
| Seed-Coder-8B-InstructModel Size=6B+, Variant=Instruct2026.03 | 62 | 66.6 | |
| Qwen3-235B-A22B-Instruct-2507Model Size=20B+, Variant=Instruct2026.03 | 62 | 89.5 | |
| Codestral-22BSize=22B2026.01 | 61.3 | 63.5 | |
| Qwen2.5-Coder-14BSize=14B2026.01 | 60.6 | 66.4 | |
| IQuest-Coder-V1-7B-ThinkingModel Size=6B+, Variant=Thinking2026.03 | 57.6 | 81.5 | |
| DeepSeek-Coder-V2-Lite-InstructModel Size=6B+, Variant=Instruct2026.03 | 57.1 | 56.2 | |
| CodeLlama-70B-InstructSize=70B2026.01 | 56.5 | 57.8 | |
| Qwen2.5-Coder-7BSize=7B2026.01 | 56.5 | 56 | |
| Stable-DiffCoder-8B-BaseSize=8B2026.01 | 53.8 | 60 | |
| DeepSeek-Coder-V2-Lite-BaseSize=2.4B/16B2026.01 | 53.4 | 46.1 | |
| DeepSeek-Coder-V2-Lite-InstructSize=2.4B/16B2026.01 | 53 | 52.9 | |
| IQuest-Coder-V1-14B-InstructModel Size=13B+, Variant=Instruct2026.03 | 52.6 | 57.6 | |
| Seed-Coder-8B-BaseSize=8B2026.01 | 52 | 54.8 | |
| DeepSeek-Coder-33B-BaseSize=33B2026.01 | 50.6 | 48.8 | |
| CodeLlama-34B-BaseSize=34B2026.01 | 49.4 | 43.9 | |
| Yi-Coder-9B-ChatSize=9B2026.01 | 47.5 | 55.6 | |
| DeepSeek-Coder-33B-InstructSize=33B2026.01 | 47.3 | 50.6 | |
| StarCoder2-15BSize=15B2026.01 | 46.1 | 47.6 | |
| IQuest-Coder-V1-7B-InstructModel Size=6B+, Variant=Instruct2026.03 | 45.8 | 54.2 | |
| StarCoder2-15B-InstructSize=15B2026.01 | 45.5 | 60.9 | |
| CodeQwen1.5-7B-ChatSize=7B2026.01 | 44 | 38.8 | |
| OpenCoder-8B-BaseSize=8B2026.01 | 43.3 | 43.9 | |
| DeepSeek-Coder-6.7B-InstructSize=6.7B2026.01 | 42.6 | 45.1 | |
| KAT-DevModel Size=20B+2026.03 | 42.5 | 65.1 | |
| OpenCoder-8B-InstructSize=8B2026.01 | 39.9 | 43 | |
| StarCoder2-7BSize=7B2026.01 | 39.5 | 35.1 | |
| DeepSeek-Coder-6.7B-BaseSize=6.7B2026.01 | 39 | 41 | |
| CodeLlama-7B-InstructSize=7B2026.01 | 36.1 | 36.2 | |
| Llama-3.1-8B-InstructSize=8B2026.01 | 35.6 | 37.8 | |
| Kimi-Dev-72BModel Size=20B+2026.03 | 33 | 64.2 | |
| Qwen3-235B-A22B-Thinking-2507Model Size=20B+, Variant=Thinking2026.03 | 15.2 | 46.9 |