Code Reasoning on MBPP (Accuracy, Token Cost)
94.55AccuracyCOPT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| COPTBackbone=Qwen3-8B, Reasoning Effort=High (Peak Accuracy)2026.05 | 94.55 | 1,997 | |
| CoTBackbone=Qwen3-8B2026.05 | 94.16 | 2,033 | |
| CoT (Greedy)Backbone=Qwen3-8B2026.05 | 91.44 | 2,724 | |
| DenserBackbone=Qwen3-14B2025.12 | 67.3 | -58.7 | |
| Self-VerificationBackbone=Qwen3-14B2025.12 | 66.1 | 142.8 | |
| Process SupervisionBackbone=Qwen3-14B2025.12 | 65.8 | 156.3 | |
| Reflection-CoTBackbone=Qwen3-14B2025.12 | 64.9 | 127.6 | |
| Tree-of-ThoughtBackbone=Qwen3-14B2025.12 | 64.5 | 345.7 | |
| TokenSkip (0.9 ratio)Backbone=Qwen3-14B2025.12 | 64.2 | -53.5 | |
| Self-ConsistencyBackbone=Qwen3-14B2025.12 | 63.4 | 289.4 | |
| Think-to-ThinkBackbone=Qwen3-14B2025.12 | 62.7 | 187.2 | |
| Chain-of-Thought (CoT)Backbone=Qwen3-14B2025.12 | 61.8 | 0 | |
| BeConciseBackbone=Qwen3-14B2025.12 | 60.1 | -54.2 | |
| AbbreWordsBackbone=Qwen3-14B2025.12 | 58.5 | -50.8 | |
| Latent Thinking OptimizationModel=Mistral-7B2025.09 | 38.8 | — | |
| Majority VotingModel=Mistral-7B2025.09 | 33.4 | — | |
| Latent Thinking OptimizationModel=Llama-2-13B2025.09 | 32.2 | — | |
| Base ModelModel=Mistral-7B2025.09 | 31.5 | — | |
| Latent Thinking OptimizationModel=OLMo-7B2025.09 | 30.8 | — | |
| Majority VotingModel=Llama-2-13B2025.09 | 26.3 | — | |
| Base ModelModel=Llama-2-13B2025.09 | 24.7 | — | |
| Base ModelModel=OLMo-7B2025.09 | 24.4 | — | |
| Majority VotingModel=OLMo-7B2025.09 | 24 | — | |
| Latent Thinking OptimizationModel=Llama-2-7B2025.09 | 23.7 | — | |
| Majority VotingModel=Llama-2-7B2025.09 | 19.3 | — | |
| Base ModelModel=Llama-2-7B2025.09 | 18.9 | — |