Execution-based Output Prediction on CRUXEval-O (200-sample, seed 42, test)
55AccuracyQwen3-14B-Base
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-14B-BaseFamily=Qwen3-Base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 55 | |
| Qwen2.5-Coder-14BFamily=Qwen2.5-Coder, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 54 | |
| Qwen2.5-14BFamily=Qwen2.5 base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 51 | |
| Qwen3-8B-BaseFamily=Qwen3-Base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 51 | |
| Qwen2.5-Coder-7B (anchor)Family=Qwen2.5-Coder, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 50.5 | |
| Qwen2.5-7BFamily=Qwen2.5 base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 49 | |
| Qwen3-4B-BaseFamily=Qwen3-Base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 45.5 | |
| DeepSeek-Coder-6.7BFamily=Cross-arch (~7B), Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 45.5 | |
| Qwen2.5-Coder-1.5BFamily=Qwen2.5-Coder, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 40 | |
| Qwen2.5-Coder-3BFamily=Qwen2.5-Coder, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 39 | |
| CodeLlama-7BFamily=Cross-arch (~7B), Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 36.5 | |
| Qwen3-1.7B-BaseFamily=Qwen3-Base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 35.5 | |
| Qwen2.5-3BFamily=Qwen2.5 base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 35 | |
| Qwen2.5-Coder-1.5B-InstructFamily=Instruct, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 34 | |
| Qwen2.5-1.5BFamily=Qwen2.5 base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 33.5 | |
| Qwen3-0.6B-BaseFamily=Qwen3-Base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 27.5 | |
| Qwen2.5-Coder-0.5B-InstructFamily=Instruct, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 27 | |
| Llama-2-7BFamily=Cross-arch (~7B), Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 17.5 | |
| Qwen2.5-0.5BFamily=Qwen2.5 base, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 7 | |
| Qwen2.5-Coder-0.5BFamily=Qwen2.5-Coder, Metric Type=pass@1, Decoding=greedy, Prompting=official direct few-shot2026.06 | 1 |