Code Generation on LiveCodeBench
90.7Pass@1Gemini-3.0 Pro
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini-3.0 ProChain-of-thought (COT)=true2025.12 | 90.7 | |
| GPT-5 HighChain-of-thought (COT)=true2025.12 | 84.5 | |
| DeepSeek-V3.2Chain-of-thought (COT)=true, thinking mode=true2025.12 | 83.3 | |
| MiniMax M2Chain-of-thought (COT)=true2025.12 | 83 | |
| Kimi-K2Chain-of-thought (COT)=true, thinking mode=true2025.12 | 82.6 | |
| DreamPRM-CodeEvaluation Protocol=Test-Time Scaling, Base Model=O4-mini-high2025.12 | 80.9 | |
| ORMEvaluation Protocol=Test-Time Scaling, Base Model=O4-mini-high2025.12 | 79.4 | |
| O4-mini-highEvaluation Protocol=Leaderboard2025.12 | 77.1 | |
| Gemini-2.5Evaluation Protocol=Leaderboard2025.12 | 72.5 | |
| Qwen3-Think-30A3BMerge Strategy=Shortest-K Merge, K=22025.12 | 72.04 | |
| O3Evaluation Protocol=Leaderboard2025.12 | 71.8 | |
| Qwen3-Think-30A3BMerge Strategy=Baseline, K=12025.12 | 69.3 | |
| DeepSeek-R1Evaluation Protocol=Leaderboard2025.12 | 68.7 | |
| Qwen3-Think-30A3BMerge Strategy=Direct-Merge, K=42025.12 | 68.69 | |
| Qwen3-Think-30A3BMerge Strategy=Direct-Merge, K=82025.12 | 68.39 | |
| Qwen3-Think-30A3BMerge Strategy=Shortest-K Merge, K=82025.12 | 67.78 | |
| Qwen3-Think-30A3BMerge Strategy=Shortest-K Merge, K=42025.12 | 67.48 | |
| Qwen3-Think-30A3BMerge Strategy=Direct-Merge, K=22025.12 | 65.65 | |
| Qwen3-4B-ThinkingMerge Strategy=Shortest-K Merge, K=42025.12 | 64.13 | |
| Claude-4.5-SonnetChain-of-thought (COT)=true2025.12 | 64 | |
| Qwen3-4B-ThinkingMerge Strategy=Shortest-K Merge, K=22025.12 | 63.83 | |
| Qwen3-4B-ThinkingMerge Strategy=Baseline, K=12025.12 | 63.53 | |
| FullKV budget=Full Cache2026.02 | 63.4 | |
| Qwen3-4B-ThinkingMerge Strategy=Direct-Merge, K=22025.12 | 62.61 | |
| Qwen3-4B-ThinkingMerge Strategy=Shortest-K Merge, K=82025.12 | 62.31 | |
| Qwen3-4B-ThinkingMerge Strategy=Direct-Merge, K=42025.12 | 62.01 | |
| ForesightKV(w/o RL)KV budget=2K2026.02 | 61.5 | |
| ForesightKVKV budget=2K2026.02 | 61.1 | |
| DeepCoder-14B-PreviewMerge Strategy=Shortest-K Merge, K=22025.12 | 61.09 | |
| Qwen3-4B-ThinkingMerge Strategy=Direct-Merge, K=82025.12 | 60.79 | |
| DeepCoder-14B-PreviewMerge Strategy=Shortest-K Merge, K=42025.12 | 59.88 | |
| DeepCoder-14B-PreviewMerge Strategy=Shortest-K Merge, K=82025.12 | 59.57 | |
| Qwen3-8BMerge Strategy=Direct-Merge, K=22025.12 | 59.57 | |
| DeepCoder-14B-PreviewMerge Strategy=Direct-Merge, K=22025.12 | 58.36 | |
| Qwen3-8BMerge Strategy=Shortest-K Merge, K=82025.12 | 58.31 | |
| Qwen3-8BMerge Strategy=Shortest-K Merge, K=22025.12 | 58.05 | |
| DeepSeek-R1-Distill-Llama-70BArchitecture=Llama, Parameter count=70B2025.01 | 57.5 | |
| DeepSeek-R1-Distill-Qwen-32BArchitecture=Qwen, Parameter count=32B2025.01 | 57.2 | |
| DeepCoder-14B-PreviewMerge Strategy=Direct-Merge, K=42025.12 | 57.14 | |
| Qwen3-8BMerge Strategy=Baseline, K=12025.12 | 57.14 | |
| Qwen3-8BMerge Strategy=Direct-Merge, K=42025.12 | 56.53 | |
| Qwen3-8BMerge Strategy=Shortest-K Merge, K=42025.12 | 56.53 | |
| DeepCoder-14B-PreviewMerge Strategy=Direct-Merge, K=82025.12 | 56.23 | |
| ForesightKV(w/o RL)KV budget=1K2026.02 | 55.7 | |
| ForesightKVKV budget=1K2026.02 | 55.7 | |
| DeepCoder-14B-PreviewMerge Strategy=Baseline, K=12025.12 | 55.32 | |
| Qwen3-8BMerge Strategy=Direct-Merge, K=82025.12 | 53.19 | |
| DeepSeek-R1-Distill-Qwen-14BArchitecture=Qwen, Parameter count=14B2025.01 | 53.1 | |
| R-KVKV budget=2K2026.02 | 52 | |
| SnapKVKV budget=2K2026.02 | 48.1 | |
| H2OKV budget=2K2026.02 | 42.5 | |
| Qwen3-Coder-30A3BMerge Strategy=Direct-Merge, K=82025.12 | 41.34 | |
| Qwen3-Coder-30A3BMerge Strategy=Direct-Merge, K=22025.12 | 39.82 | |
| Qwen3-Coder-30A3BMerge Strategy=Shortest-K Merge, K=82025.12 | 39.82 | |
| DeepSeek-R1-Distill-Llama-8BArchitecture=Llama, Parameter count=8B2025.01 | 39.6 | |
| Qwen3-Coder-30A3BMerge Strategy=Shortest-K Merge, K=22025.12 | 39.21 | |
| Claude-3.5-Sonnet-10222025.01 | 38.9 | |
| Qwen3-Coder-30A3BMerge Strategy=Direct-Merge, K=42025.12 | 38.3 | |
| Qwen3-Coder-30A3BMerge Strategy=Shortest-K Merge, K=42025.12 | 38.3 | |
| Qwen3-Coder-30A3BMerge Strategy=Baseline, K=12025.12 | 37.69 | |
| DeepSeek-R1-Distill-Qwen-7BArchitecture=Qwen, Parameter count=7B2025.01 | 37.6 | |
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 34.6 | |
| R-KVKV budget=1K2026.02 | 34.4 | |
| GPT-4o-05132025.01 | 32.9 | |
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 31.6 | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 31.4 | |
| H2OKV budget=1K2026.02 | 28.3 | |
| DS-Coder-V2-InstructSize Category=32B+ LLMs2025.12 | 27.9 | |
| SnapKVKV budget=1K2026.02 | 27 | |
| DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.12 | 24.3 | |
| Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.12 | 23.4 | |
| OpenCoder-8B-InstructSize Category=6B+ LLMs2025.12 | 23.2 | |
| UCoder-7BSize Category=6B+ LLMs2025.12 | 22.9 | |
| UCoder-32BSize Category=32B+ LLMs2025.12 | 21.4 | |
| DS-Coder-33B-InstructSize Category=32B+ LLMs2025.12 | 21.3 | |
| UCoder-14BSize Category=13B+ LLMs2025.12 | 20.6 | |
| Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.12 | 18.2 | |
| DeepSeek-R1-Distill-Qwen-1.5BArchitecture=Qwen, Parameter count=1.5B2025.01 | 16.9 | |
| DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.12 | 15.5 | |
| Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.12 | 12.1 | |
| CodeLlama-34B-InstructSize Category=32B+ LLMs2025.12 | 8.4 | |
| CodeLlama-7B-InstructSize Category=6B+ LLMs2025.12 | 7.1 | |
| CodeLlama-13B-InstructSize Category=13B+ LLMs2025.12 | 6.1 | |
| DeepSeek-R1Architecture=MoE, Activated Params=37B, Total Params=671B, prompting=COT2025.01 | 0.659 | |
| OpenAI-o1-1217prompting=COT2025.01 | 0.634 | |
| OpenAI-o1-miniprompting=COT2025.01 | 0.538 | |
| Claude-3.5-Sonnet-1022prompting=COT2025.01 | 0.389 | |
| DeepSeek-V3Architecture=MoE, Activated Params=37B, Total Params=671B, prompting=COT2025.01 | 0.362 | |
| GPT-4o-0513prompting=COT2025.01 | 0.329 |