List your company
ResearchFormerly WizWand

Code Generation on BigCodeBench-Instruct Hard

28.4Pass@1o1
2.49.1515.922.65Dec 5, 2025Dec 21, 2025Jan 6, 2026Jan 23, 2026Feb 8, 2026Feb 24, 2026Mar 13, 2026
Updated 7mo ago

Evaluation results

MethodLinks
o12026.0328.4
Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.1227
DeepSeek-V2.5-238B2026.0327
Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.1225.7
GPT-4o-2024-08-06Size Category=Closed-APIs2025.1225
EVOLVECODER-4B (r3)Refinement round=r3, Backbone=Qwen3-4B (Thinking)2026.0325
DS-Coder-V2-InstructSize Category=32B+ LLMs2025.1224.3
EVOLVECODER-4B (r2)Refinement round=r2, Backbone=Qwen3-4B (Thinking)2026.0324.3
Critique-Coder-4B2026.0323
EVOLVECODER-4B (r0)Refinement round=r0, Backbone=Qwen3-4B (Thinking)2026.0323
EVOLVECODER-4B (r1)Refinement round=r1, Backbone=Qwen3-4B (Thinking)2026.0323
Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.1222.2
DeepSeek-R1-Distill-14B2026.0320.9
Qwen3-4B (Thinking)Thinking-mode sampling=true2026.0320.9
AceCoder-7B2026.0319.6
OpenCoder-8B-InstructSize Category=6B+ LLMs2025.1218.2
Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.1218.2
DeepCoder-14B2026.0318.2
DS-Coder-33B-InstructSize Category=32B+ LLMs2025.1217.6
UCoder-32BSize Category=32B+ LLMs2025.1217.6
DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.1216.2
UCoder-14BSize Category=13B+ LLMs2025.1216.2
NonDPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=None2025.1216.2
UCoder-7BSize Category=6B+ LLMs2025.1215.5
Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.1211.5
NonDPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=None2025.1210.8
PrivCodeBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.1210.8
DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.1210.1
CodeLlama-13B-InstructSize Category=13B+ LLMs2025.129.5
NonDPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=None2025.129.5
PrivCodeBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.129.5
DP-AdapterBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.129.5
JFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.129.2
CodeLlama-34B-InstructSize Category=32B+ LLMs2025.128.8
DPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.128.8
NonDPFTBackbone=CodeGemma-7B, Privacy Epsilon=None2025.128.1
JFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.127.9
PrivCodeBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.127.4
DPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.125.4
PrivCodeBackbone=CodeGemma-7B, Privacy Epsilon=42025.125.4
JFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.125.3
DPFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.125.1
DP-AdapterBackbone=CodeGemma-7B, Privacy Epsilon=42025.124.8
JFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.124.8
DPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.124.1
DP-AdapterBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.123.9
DP-AdapterBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.123.8
CodeLlama-7B-InstructSize Category=6B+ LLMs2025.123.4