Code Reasoning on CRUXEval-O
83.5AccuracyKimi-K2 Base
Evaluation results
| Method | Links | |
|---|---|---|
| Kimi-K2 Base# Shots=1-shot, # Activated Params=32B, # Total Params=1043B2026.01 | 83.5 | |
| JCN=24, Trace Generator=Qwen3-Coder-30B2026.05 | 80.5 | |
| JCN=16, Trace Generator=Qwen3-Coder-30B2026.05 | 79.9 | |
| MiMo-V2-Flash Base# Shots=1-shot, # Activated Params=15B, # Total Params=309B2026.01 | 79.1 | |
| JCN=8, Trace Generator=Qwen3-Coder-30B2026.05 | 78.7 | |
| WSCN=16, Trace Generator=Qwen3-Coder-30B2026.05 | 78.1 | |
| WSCN=24, Trace Generator=Qwen3-Coder-30B2026.05 | 78.1 | |
| BoNN=16, Trace Generator=Qwen3-Coder-30B2026.05 | 78 | |
| BoNN=24, Trace Generator=Qwen3-Coder-30B2026.05 | 78 | |
| WSCN=8, Trace Generator=Qwen3-Coder-30B2026.05 | 77.8 | |
| JCN=4, Trace Generator=Qwen3-Coder-30B2026.05 | 77.3 | |
| BoNN=8, Trace Generator=Qwen3-Coder-30B2026.05 | 77 | |
| WSCN=4, Trace Generator=Qwen3-Coder-30B2026.05 | 76.9 | |
| DeepSeek-V3.1 Base# Shots=1-shot, # Activated Params=37B, # Total Params=671B2026.01 | 76.4 | |
| SCN=24, Trace Generator=Qwen3-Coder-30B2026.05 | 76.4 | |
| SCN=16, Trace Generator=Qwen3-Coder-30B2026.05 | 76.3 | |
| BoNN=4, Trace Generator=Qwen3-Coder-30B2026.05 | 76.2 | |
| SCN=8, Trace Generator=Qwen3-Coder-30B2026.05 | 76 | |
| Self-Cert.N=24, Trace Generator=Qwen3-Coder-30B2026.05 | 76 | |
| DeepConfN=24, Trace Generator=Qwen3-Coder-30B2026.05 | 75.8 | |
| Self-Cert.N=16, Trace Generator=Qwen3-Coder-30B2026.05 | 75.6 | |
| DeepConfN=16, Trace Generator=Qwen3-Coder-30B2026.05 | 75.6 | |
| DeepSeek-V3.2 Exp Base# Shots=1-shot, # Activated Params=37B, # Total Params=671B2026.01 | 74.9 | |
| SCN=4, Trace Generator=Qwen3-Coder-30B2026.05 | 74.7 | |
| Self-Cert.N=8, Trace Generator=Qwen3-Coder-30B2026.05 | 74.7 | |
| DeepConfN=8, Trace Generator=Qwen3-Coder-30B2026.05 | 74.7 | |
| Self-Cert.N=4, Trace Generator=Qwen3-Coder-30B2026.05 | 73.5 | |
| DeepConfN=4, Trace Generator=Qwen3-Coder-30B2026.05 | 71.9 | |
| Pass@1N=4, Trace Generator=Qwen3-Coder-30B2026.05 | 71.3 | |
| Pass@1N=8, Trace Generator=Qwen3-Coder-30B2026.05 | 71.3 | |
| Pass@1N=16, Trace Generator=Qwen3-Coder-30B2026.05 | 71.3 | |
| Pass@1N=24, Trace Generator=Qwen3-Coder-30B2026.05 | 71.3 | |
| INTUITORBackbone=Qwen3-14B, Inference Template=chat inference template2025.05 | 67.7 | |
| Qwen3-14BBackbone=Qwen3-14B, Inference Template=chat inference template2025.05 | 66.3 | |
| INTUITORBackbone=Qwen2.5-7B, Inference Template=chat inference template2025.05 | 57.4 | |
| MPModel=Qwen-3-8B, Prompting Strategy=Metacognitive Prompting2026.02 | 56.5 | |
| INTUITORBackbone=Qwen2.5-14B, Inference Template=chat inference template2025.05 | 56 | |
| Ann BrownModel=Qwen-3-8B, Prompting Strategy=Ann Brown Prompting2026.02 | 55.88 | |
| CoTModel=Qwen-3-8B, Prompting Strategy=Chain-of-Thought2026.02 | 55.75 | |
| StdModel=Qwen-3-8B, Prompting Strategy=Standard2026.02 | 55.5 | |
| GRPOBackbone=Qwen2.5-7B, Inference Template=chat inference template2025.05 | 53.8 | |
| GRPOBackbone=Qwen2.5-14B, Inference Template=chat inference template2025.05 | 52 | |
| Qwen2.5-14BBackbone=Qwen2.5-14B, Inference Template=chat inference template2025.05 | 49.1 | |
| Mellum 2Parameters=2.5B/12B2026.05 | 43.9 | |
| Qwen3-4BParameters=4B2026.05 | 43.5 | |
| Qwen3.5-4BParameters=4B2026.05 | 43.2 | |
| Qwen2.5-7BParameters=7B2026.05 | 42.9 | |
| INTUITORModel=Qwen2.5-3B, Training Data=MATH, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 41.6 | |
| INTUITOR-CodeModel=Qwen2.5-3B, Training Data=Codeforces, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 41.1 | |
| OLMo-3-7BParameters=7B2026.05 | 36.6 | |
| GRPOModel=Qwen2.5-3B, Training Data=MATH, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 34.1 | |
| GRPOModel=Qwen2.5-1.5B, Training Data=MATH, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 32.8 | |
| GRPO-PVModel=Qwen2.5-3B, Training Data=MATH, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 29.9 | |
| INTUITORModel=Qwen2.5-1.5B, Training Data=MATH, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 29.6 | |
| CoTModel=Llama-3-8B, Prompting Strategy=Chain-of-Thought2026.02 | 25.62 | |
| BaseModel=Qwen2.5-3B, Training Data=-, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 23.6 | |
| Qwen2.5-7BBackbone=Qwen2.5-7B, Inference Template=chat inference template2025.05 | 17.8 | |
| StdModel=Llama-3-8B, Prompting Strategy=Standard2026.02 | 5.25 | |
| Ann BrownModel=Llama-3-8B, Prompting Strategy=Ann Brown Prompting2026.02 | 5.12 | |
| MPModel=Llama-3-8B, Prompting Strategy=Metacognitive Prompting2026.02 | 3.38 | |
| BaseModel=Qwen2.5-1.5B, Training Data=-, Inference template=chat inference template, Decoding strategy=Greedy2025.05 | 0 |