List your company
ResearchFormerly WizWand

Code Generation on HumanEval (pass@1, Final Gap)

96.95pass@1Qwen3 Model
58.7882468.6956278.60388.51038Feb 8, 2025May 2, 2025Jul 25, 2025Oct 16, 2025Jan 8, 2026Apr 1, 2026Jun 24, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Qwen3 ModelParameters=235B-25072025.0996.95——
Uno-Orchestra2026.0593.1——
TRACE-LFLLM=Qwen3-32B2026.0590.9——
Pioneer AgentModel=Qwen3-8B, System Strategy=Agent, Checkpoint Stage=Best2026.0490.317.8—
CodeSimLLM=LLaMa3.1-70B2025.0290.2——
Qwen3 ModelParameters=8B2025.0989.63——
AgentOrchestra2026.0589.2——
Pioneer AgentModel=Qwen3-8B, System Strategy=Agent, Checkpoint Stage=C12026.048917.8—
xRouter2026.0588.1——
FULL TOKENSBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=100.02026.0687.8——
CODEBLOCKBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.0687.8——
BASEBase model=Qwen2.5-Coder-7B-Instruct2026.0687.2——
Qwen3 ModelParameters=32B2025.0987.2——
CODEBLOCKBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.0686.6——
CLAMBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=3.52026.0686.6——
DS2Base model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=4.62026.0686——
MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.32026.0685.98——
RANDOM SELECTIONBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.0685.4——
DS2Base model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=4.62026.0685.4——
CLAMBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=3.52026.0685.4——
MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.52026.0685.37——
RANDOM SELECTIONBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.0685.1——
Qwen2.5 7BModel Architecture=AR, Instruction Tuning=true2026.0684.8——
Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.52026.0684.76——
MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.72026.0684.76——
AOrchestra2026.0584.2——
BASEBase model=Qwen2.5-Coder-3B-Instruct2026.0684.2——
TOKEN CLEANINGBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.72026.0684.2——
Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.72026.0684.15——
Qwen3 ModelParameters=4B-25072025.0983.54——
Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.32026.0682.93——
CodeSimLLM=Gemma2-9B2025.0282.9——
AR-OPD2026.0682.6——
FULL TOKENSBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=100.02026.0682.3——
TOKEN CLEANINGBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.72026.0682.3——
Single LLMBackbone=Qwen2.5-32B-Instruct2026.0681.71——
AAPAParameters=4B2025.0981.71——
TRACE-CSLLM=Qwen3-32B2026.0581.1——
CodeSimLLM=LLaMa3.1-8B2025.0279.9——
Partial OPD2026.0679.4——
Base2026.0679.2——
Full OPD2026.0678.8——
MOCEdge Density=0.5, Backbone=Gemma-2-27B-Instruct2026.0678.66——
MOCEdge Density=0.3, Backbone=Gemma-2-27B-Instruct2026.0678.05——
MOCEdge Density=0.7, Backbone=Gemma-2-27B-Instruct2026.0678.05——
Vanilla MASEdge Density=0.5, Backbone=Gemma-2-27B-Instruct2026.0677.44——
Vanilla MASEdge Density=0.7, Backbone=Gemma-2-27B-Instruct2026.0677.44——
Vanilla MASEdge Density=0.3, Backbone=Gemma-2-27B-Instruct2026.0676.83——
MOCEdge Density=1.0, Backbone=Gemma-2-27B-Instruct2026.0676.83——
SFT2026.0676.8——
Vanilla MASEdge Density=1.0, Backbone=Gemma-2-27B-Instruct2026.0676.22——
Qwen3 ModelParameters=4B2025.0975.61——
CoTLLM=LLaMa3.1-70B2025.0275.6——
TRACE-CSLLM=Qwen2.5-14B2026.0575.6——
CodeSimLLM=Mixtral8x7B2025.0275——
Sequential Fine-tuningLLM=Qwen3-32B2026.0575——
Sequential LoRALLM=Qwen3-32B2026.0575——
Naive BaselineModel=Qwen3-8B, System Strategy=Naive, Checkpoint Stage=C12026.0474.417.8—
ReflexionLLM=LLaMa3.1-70B2025.0273.8——
Joint Fine-tuningLLM=Qwen3-32B2026.0573.8——
TRACE-LFLLM=Qwen2.5-14B2026.0573.2——
Single LLMBackbone=Gemma-2-27B-Instruct2026.0673.17——
AC-ODM-410Mproxy_model=410M Pythia2025.0572.644——
DreamCoder-7B-InstructEnsemble Strategy=Single Model2026.0672.56——
DreamCoder + DiffuCoderEnsemble Strategy=Intermediate-generation Ensemble, Scoring Function=Token change count2026.0672.56——
DreamCoder + DiffuCoderEnsemble Strategy=Intermediate-generation Ensemble, Scoring Function=Top-1 probability2026.0672.56——
Naive BaselineModel=Qwen3-8B, System Strategy=Naive, Checkpoint Stage=C42026.0472.517.8—
ColdStart-LLM2026.0572.4——
Qwen2.5-Coder-7B w/ CADFTModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=CADFT2026.0471.3—61.5
DiffuCoder-7B-InstructEnsemble Strategy=Single Model2026.0670.12——
DMTLLM=Qwen3-32B2026.0568.9——
Qwen2.5-Coder-7B w/ DFTModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=DFT2026.0467.7—59.8
BaselineModel=Llama 3.1-8B2026.0267.07——
Qwen3 ModelParameters=1.7B2025.0967.07——
TACBackbone=Qwen3-1.7B, Curriculum Strategy=TAC2026.0666.7——
InnerQBaseModel=Llama 3.1-8B2026.0266.46——
iLLaDA 8BModel Architecture=Diffusion, Instruction Tuning=true2026.0665.9——
RouterDCModel scale=1.5B, Evaluation mode=0-shot2026.0665.24——
DLLGModel scale=1.5B, Evaluation mode=0-shot2026.0665.24——
RandomBackbone=Qwen3-1.7B, Curriculum Strategy=Random2026.0665.2——
M2OBackbone=Qwen3-1.7B, Curriculum Strategy=M2O2026.0665——
SECBackbone=Qwen3-1.7B, Curriculum Strategy=SEC2026.0664.8——
InnerQHybridModel=Llama 3.1-8B2026.0264.63——
InnerQSmallModel=Llama 3.1-8B2026.0264.63——
Qwen2.5-Coder-1.5B-InstructModel scale=1.5B, Evaluation mode=0-shot2026.0664.63——
KIVIModel=Llama 3.1-8B2026.0264.02——
KIVISinkModel=Llama 3.1-8B2026.0264.02——
Pack of LLMsModel scale=1.5B, Evaluation mode=0-shot2026.0664.02——
DirectLLM=Gemma2-9B2025.0264——
MSA-PTTraining Strategy=from-scratch sparse pretraining2026.0664——
TurboQuantModel=Llama 3.1-8B2026.0263.42——
Token Maj-VotingModel scale=1.5B, Evaluation mode=0-shot2026.0662.8——
ReflexionLLM=Gemma2-9B2025.0262.2——
Qwen2.5-Coder-7BModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=None2026.0462.2—53
Entropy WeightingModel scale=1.5B, Evaluation mode=0-shot2026.0662.19——
TRACE-CSLLM=LLaMA3-8B2026.0561.6——
UniTeModel scale=1.5B, Evaluation mode=0-shot2026.0661.59——
FullTraining Strategy=Full-Attention baseline2026.0661——
GaCModel scale=1.5B, Evaluation mode=0-shot2026.0660.98——
AC-ODM2025.0560.256——