Code Generation on HumanEval (pass@1, Final Gap)
96.95pass@1Qwen3 Model
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Qwen3 ModelParameters=235B-25072025.09 | 96.95 | — | — | |
| Uno-Orchestra2026.05 | 93.1 | — | — | |
| TRACE-LFLLM=Qwen3-32B2026.05 | 90.9 | — | — | |
| Pioneer AgentModel=Qwen3-8B, System Strategy=Agent, Checkpoint Stage=Best2026.04 | 90.3 | 17.8 | — | |
| CodeSimLLM=LLaMa3.1-70B2025.02 | 90.2 | — | — | |
| Qwen3 ModelParameters=8B2025.09 | 89.63 | — | — | |
| AgentOrchestra2026.05 | 89.2 | — | — | |
| Pioneer AgentModel=Qwen3-8B, System Strategy=Agent, Checkpoint Stage=C12026.04 | 89 | 17.8 | — | |
| xRouter2026.05 | 88.1 | — | — | |
| FULL TOKENSBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=100.02026.06 | 87.8 | — | — | |
| CODEBLOCKBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.06 | 87.8 | — | — | |
| BASEBase model=Qwen2.5-Coder-7B-Instruct2026.06 | 87.2 | — | — | |
| Qwen3 ModelParameters=32B2025.09 | 87.2 | — | — | |
| CODEBLOCKBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.06 | 86.6 | — | — | |
| CLAMBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=3.52026.06 | 86.6 | — | — | |
| DS2Base model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=4.62026.06 | 86 | — | — | |
| MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.32026.06 | 85.98 | — | — | |
| RANDOM SELECTIONBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.06 | 85.4 | — | — | |
| DS2Base model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=4.62026.06 | 85.4 | — | — | |
| CLAMBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=3.52026.06 | 85.4 | — | — | |
| MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.52026.06 | 85.37 | — | — | |
| RANDOM SELECTIONBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.06 | 85.1 | — | — | |
| Qwen2.5 7BModel Architecture=AR, Instruction Tuning=true2026.06 | 84.8 | — | — | |
| Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.52026.06 | 84.76 | — | — | |
| MOCBackbone=Qwen2.5-32B-Instruct, Edge Density=0.72026.06 | 84.76 | — | — | |
| AOrchestra2026.05 | 84.2 | — | — | |
| BASEBase model=Qwen2.5-Coder-3B-Instruct2026.06 | 84.2 | — | — | |
| TOKEN CLEANINGBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.72026.06 | 84.2 | — | — | |
| Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.72026.06 | 84.15 | — | — | |
| Qwen3 ModelParameters=4B-25072025.09 | 83.54 | — | — | |
| Vanilla MASBackbone=Qwen2.5-32B-Instruct, Edge Density=0.32026.06 | 82.93 | — | — | |
| CodeSimLLM=Gemma2-9B2025.02 | 82.9 | — | — | |
| AR-OPD2026.06 | 82.6 | — | — | |
| FULL TOKENSBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=100.02026.06 | 82.3 | — | — | |
| TOKEN CLEANINGBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.72026.06 | 82.3 | — | — | |
| Single LLMBackbone=Qwen2.5-32B-Instruct2026.06 | 81.71 | — | — | |
| AAPAParameters=4B2025.09 | 81.71 | — | — | |
| TRACE-CSLLM=Qwen3-32B2026.05 | 81.1 | — | — | |
| CodeSimLLM=LLaMa3.1-8B2025.02 | 79.9 | — | — | |
| Partial OPD2026.06 | 79.4 | — | — | |
| Base2026.06 | 79.2 | — | — | |
| Full OPD2026.06 | 78.8 | — | — | |
| MOCEdge Density=0.5, Backbone=Gemma-2-27B-Instruct2026.06 | 78.66 | — | — | |
| MOCEdge Density=0.3, Backbone=Gemma-2-27B-Instruct2026.06 | 78.05 | — | — | |
| MOCEdge Density=0.7, Backbone=Gemma-2-27B-Instruct2026.06 | 78.05 | — | — | |
| Vanilla MASEdge Density=0.5, Backbone=Gemma-2-27B-Instruct2026.06 | 77.44 | — | — | |
| Vanilla MASEdge Density=0.7, Backbone=Gemma-2-27B-Instruct2026.06 | 77.44 | — | — | |
| Vanilla MASEdge Density=0.3, Backbone=Gemma-2-27B-Instruct2026.06 | 76.83 | — | — | |
| MOCEdge Density=1.0, Backbone=Gemma-2-27B-Instruct2026.06 | 76.83 | — | — | |
| SFT2026.06 | 76.8 | — | — | |
| Vanilla MASEdge Density=1.0, Backbone=Gemma-2-27B-Instruct2026.06 | 76.22 | — | — | |
| Qwen3 ModelParameters=4B2025.09 | 75.61 | — | — | |
| CoTLLM=LLaMa3.1-70B2025.02 | 75.6 | — | — | |
| TRACE-CSLLM=Qwen2.5-14B2026.05 | 75.6 | — | — | |
| CodeSimLLM=Mixtral8x7B2025.02 | 75 | — | — | |
| Sequential Fine-tuningLLM=Qwen3-32B2026.05 | 75 | — | — | |
| Sequential LoRALLM=Qwen3-32B2026.05 | 75 | — | — | |
| Naive BaselineModel=Qwen3-8B, System Strategy=Naive, Checkpoint Stage=C12026.04 | 74.4 | 17.8 | — | |
| ReflexionLLM=LLaMa3.1-70B2025.02 | 73.8 | — | — | |
| Joint Fine-tuningLLM=Qwen3-32B2026.05 | 73.8 | — | — | |
| TRACE-LFLLM=Qwen2.5-14B2026.05 | 73.2 | — | — | |
| Single LLMBackbone=Gemma-2-27B-Instruct2026.06 | 73.17 | — | — | |
| AC-ODM-410Mproxy_model=410M Pythia2025.05 | 72.644 | — | — | |
| DreamCoder-7B-InstructEnsemble Strategy=Single Model2026.06 | 72.56 | — | — | |
| DreamCoder + DiffuCoderEnsemble Strategy=Intermediate-generation Ensemble, Scoring Function=Token change count2026.06 | 72.56 | — | — | |
| DreamCoder + DiffuCoderEnsemble Strategy=Intermediate-generation Ensemble, Scoring Function=Top-1 probability2026.06 | 72.56 | — | — | |
| Naive BaselineModel=Qwen3-8B, System Strategy=Naive, Checkpoint Stage=C42026.04 | 72.5 | 17.8 | — | |
| ColdStart-LLM2026.05 | 72.4 | — | — | |
| Qwen2.5-Coder-7B w/ CADFTModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=CADFT2026.04 | 71.3 | — | 61.5 | |
| DiffuCoder-7B-InstructEnsemble Strategy=Single Model2026.06 | 70.12 | — | — | |
| DMTLLM=Qwen3-32B2026.05 | 68.9 | — | — | |
| Qwen2.5-Coder-7B w/ DFTModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=DFT2026.04 | 67.7 | — | 59.8 | |
| BaselineModel=Llama 3.1-8B2026.02 | 67.07 | — | — | |
| Qwen3 ModelParameters=1.7B2025.09 | 67.07 | — | — | |
| TACBackbone=Qwen3-1.7B, Curriculum Strategy=TAC2026.06 | 66.7 | — | — | |
| InnerQBaseModel=Llama 3.1-8B2026.02 | 66.46 | — | — | |
| iLLaDA 8BModel Architecture=Diffusion, Instruction Tuning=true2026.06 | 65.9 | — | — | |
| RouterDCModel scale=1.5B, Evaluation mode=0-shot2026.06 | 65.24 | — | — | |
| DLLGModel scale=1.5B, Evaluation mode=0-shot2026.06 | 65.24 | — | — | |
| RandomBackbone=Qwen3-1.7B, Curriculum Strategy=Random2026.06 | 65.2 | — | — | |
| M2OBackbone=Qwen3-1.7B, Curriculum Strategy=M2O2026.06 | 65 | — | — | |
| SECBackbone=Qwen3-1.7B, Curriculum Strategy=SEC2026.06 | 64.8 | — | — | |
| InnerQHybridModel=Llama 3.1-8B2026.02 | 64.63 | — | — | |
| InnerQSmallModel=Llama 3.1-8B2026.02 | 64.63 | — | — | |
| Qwen2.5-Coder-1.5B-InstructModel scale=1.5B, Evaluation mode=0-shot2026.06 | 64.63 | — | — | |
| KIVIModel=Llama 3.1-8B2026.02 | 64.02 | — | — | |
| KIVISinkModel=Llama 3.1-8B2026.02 | 64.02 | — | — | |
| Pack of LLMsModel scale=1.5B, Evaluation mode=0-shot2026.06 | 64.02 | — | — | |
| DirectLLM=Gemma2-9B2025.02 | 64 | — | — | |
| MSA-PTTraining Strategy=from-scratch sparse pretraining2026.06 | 64 | — | — | |
| TurboQuantModel=Llama 3.1-8B2026.02 | 63.42 | — | — | |
| Token Maj-VotingModel scale=1.5B, Evaluation mode=0-shot2026.06 | 62.8 | — | — | |
| ReflexionLLM=Gemma2-9B2025.02 | 62.2 | — | — | |
| Qwen2.5-Coder-7BModel Backbone=Qwen2.5-Coder-7B, Fine-tuning Strategy=None2026.04 | 62.2 | — | 53 | |
| Entropy WeightingModel scale=1.5B, Evaluation mode=0-shot2026.06 | 62.19 | — | — | |
| TRACE-CSLLM=LLaMA3-8B2026.05 | 61.6 | — | — | |
| UniTeModel scale=1.5B, Evaluation mode=0-shot2026.06 | 61.59 | — | — | |
| FullTraining Strategy=Full-Attention baseline2026.06 | 61 | — | — | |
| GaCModel scale=1.5B, Evaluation mode=0-shot2026.06 | 60.98 | — | — | |
| AC-ODM2025.05 | 60.256 | — | — |