Coding Reasoning on HumanEval
96.34Accuracy (%)COPT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| COPTBackbone=Qwen3-8B, Reasoning Effort=High (Peak Accuracy)2026.05 | 96.34 | 1,842 | — | |
| COPTBackbone=Qwen3-8B, Reasoning Effort=Token-saving / Matched Accuracy2026.05 | 94.51 | 1,023 | — | |
| CoT (Greedy)Backbone=Qwen3-8B2026.05 | 93.9 | 2,627 | — | |
| ASAGModel=Qwen3-14B2026.06 | 93.9 | 1,025 | 32.1 | |
| ASAGModel=Qwen3-32B2026.06 | 93.9 | 1,126 | 35 | |
| VanillaModel=Qwen3-32B2026.06 | 93.3 | 3,216 | 100 | |
| ASAGModel=Qwen3-4B2026.06 | 92.7 | 1,249 | 32.7 | |
| ASAGModel=Qwen3-8B2026.06 | 92.7 | 1,002 | 26.3 | |
| VanillaModel=Qwen3-14B2026.06 | 92.7 | 3,198 | 100 | |
| CoTBackbone=Qwen3-8B2026.05 | 92.68 | 2,368 | — | |
| VanillaModel=Qwen3-4B2026.06 | 91.5 | 3,821 | 100 | |
| VanillaModel=Qwen3-8B2026.06 | 90.2 | 3,812 | 100 | |
| ASAGModel=DeepSeek-R1-Distill-Llama-8B2026.06 | 82.3 | 1,465 | 23.6 | |
| VanillaModel=DeepSeek-R1-Distill-Llama-8B2026.06 | 80.5 | 6,214 | 100 | |
| VanillaModel=DeepSeek-R1-Distill-Qwen-7B2026.06 | 78.6 | 5,739 | 100 | |
| ASAGModel=DeepSeek-R1-Distill-Qwen-7B2026.06 | 78.6 | 1,257 | 21.9 |