List your company
ResearchFormerly WizWand

Coding on HumanEval

98.17Pass@1CompassMax-V3-Thinking
45.577259.231172.88586.5389Jul 15, 2024Nov 2, 2024Feb 20, 2025Jun 10, 2025Sep 28, 2025Jan 16, 2026May 6, 2026
Updated 5mo ago

Evaluation results

MethodLinks
CompassMax-V3-Thinking2025.1298.17
DeepSeek-R12025.1296.95
SwiRBackbone=Qwen3-8B2025.1095.73
CoT (Greedy)Backbone=Qwen3-8B, decoding=Greedy2025.1093.9
CoTBackbone=Qwen3-8B2025.1092.68
Soft ThinkingBackbone=Qwen3-8B2025.1092.07
MT (Teacher)Teacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0491.5
GPT-4o2024.1090.2
Llama-3.1-405BModel Type=Instruct, Parameters=405B2025.0289
Qwen2.5-VL-72BModel Type=Instruct, Parameters=72B2025.0287.8
GPT-4o mini2024.1087.2
Qwen2.5-72BModel Type=Instruct, Parameters=72B2025.0286.6
Qwen2-72B-InstructType=Instruction-tuned2024.0786
Qwen2-72BModel Type=Instruct, Parameters=72B2025.0286
CompassMax-V32025.1284.76
Gemini-1.5 Pro2024.1084.1
UltraMix-190kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-190k2025.1182.27
Llama-3-70B-InstructType=Instruction-tuned2024.0781.7
UltraMix-187kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-187k2025.1181.1
Llama-3.1-70BModel Type=Instruct, Parameters=70B2025.0280.5
TuluDPOBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.1180.49
HPDTeacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0479.3
UltraMix-170kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-170k2025.1178.05
KDTeacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0477.4
JSDTeacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0477.4
RKLDTeacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0476.8
MT (Teacher)Teacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0476.2
Qwen1.5-110B-ChatType=Instruction-tuned2024.0774.4
Gemini-1.5 Flash2024.1074.3
Mixtral-8x22B-InstructType=Instruction-tuned2024.0773.8
SFTTeacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0473.8
BaseBackbone=Qwen2.5-3B-Instruct2026.0573.8
BaseBackbone=Qwen2.5-3B-Instruct2026.0573.8
BaseBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Pre-trained2026.0573.8
ARIA2024.1073.2
LangMARLCategory=Ours2026.0473.2
CoT2-MetaStrategy=Ours (CoT2-Meta), Inference Budget=C=162026.0372.8
SFTBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=SFT2025.1172.66
Llama3.2-11B2024.1072.6
Pixtral-12B2024.1072
Qwen1.5-72B-ChatType=Instruction-tuned2024.0771.3
MS (Student)Teacher-Student Model Configuration=Qwen-Coder (7B -> 1.5B)2026.0471.3
STMBackbone=Qwen2.5-3B-Instruct2026.0571.3
Self-SFTBackbone=Qwen2.5-3B-Instruct2026.0571.3
Iter-SFTBackbone=Qwen2.5-3B-Instruct2026.0571.3
Low-SFTBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 3: iGSM -> MedCalc -> IFEval2026.0571.3
Anchored LearningBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 3: iGSM -> MedCalc -> IFEval2026.0570.7
ReST-MCTS*Strategy=ReST-MCTS*, Inference Budget=C=162026.0370.4
ReflexionCategory=Self-Evolving2026.0470.1
Anchored LearningBackbone=Qwen2.5-3B-Instruct2026.0570.1
Anchored LearningBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 1: iGSM2026.0570.1
HPDTeacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0469.5
UltraMix-190kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-190k2025.1169.05
TextGradCategory=Self-Evolving2026.0468.9
Self-sftBackbone=Qwen2.5-3B-Instruct2026.0568.9
Low-SFTBackbone=Qwen2.5-3B-Instruct2026.0568.9
Low-SFTBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 1: iGSM2026.0568.9
Anchored LearningBackbone=Qwen2.5-3B-Instruct2026.0568.3
Vanilla ToTStrategy=Vanilla ToT, Inference Budget=C=162026.0368.2
UltraMix-187kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-187k2025.1168.06
TuluDPOBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.1167.24
JSDTeacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0467.1
GPT-4V2024.1067
DSPyCategory=Self-Evolving2026.0466.7
Low-SFTBackbone=Qwen2.5-3B-Instruct2026.0566.5
UltraMix-170kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-170k2025.1165.61
Best-of-16Strategy=Best-of-16, Inference Budget=C=162026.0365.4
KDTeacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0465.2
Youtu-LLMSize=2B, Type=Base, Prompt Setting=0-shot2025.1264.6
STMBackbone=Qwen2.5-3B-Instruct2026.0564.6
Low-SFTBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 2: iGSM -> MedCalc2026.0564.6
SymbolicCategory=Self-Evolving2026.0464.5
Iter-SFTBackbone=Qwen2.5-3B-Instruct2026.0564
AutoPECategory=Self-Evolving2026.0463.5
MS (Student)Teacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0462.8
ProSeCo SamplingCorrector Sampling=true, Backbone=LLaDA-Base 8B, Number of shots=02026.0262.2
RKLDTeacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0461.6
SFTTeacher-Student Model Configuration=DS-Coder (6.7B -> 1.3B)2026.0461
Anchored LearningBackbone=Qwen2.5-3B-Instruct, Incremental Learning Stage=Stage 2: iGSM -> MedCalc2026.0561
Greedy CoTStrategy=Greedy CoT, Inference Budget=C=162026.0360.5
DFTBackbone=Qwen2.5-3B-Instruct2026.0560.4
AgentsCategory=Static-Prompting2026.0459.5
CoTCategory=Static-Prompting2026.0459.2
Llama3.1-InstructCorrector Sampling=false, Model scale=8B, Number of shots=02026.0258.54
SFTBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=SFT2025.1157.93
Iter-SFTBackbone=Llama-3.2-3B-Instruct2026.0557.7
Qwen3Size=4B, Type=Base, Prompt Setting=0-shot2025.1257.6
Anchored LearningBackbone=Llama-3.2-3B-Instruct2026.0556.1
BaseBackbone=Llama-3.2-3B-Instruct2026.0554.9
Low-SFTBackbone=Llama-3.2-3B-Instruct2026.0553.7
STMBackbone=Llama-3.2-3B-Instruct2026.0553.7
Self-SFTBackbone=Llama-3.2-3B-Instruct2026.0553.7
ProSeCo SFTCorrector Sampling=false, Backbone=LLaDA-Base 8B, Number of shots=02026.0252.44
CodePrefBackbone=Apertus-8B-SFT2025.1151.24
Qwen3Size=1.7B, Type=Base, Prompt Setting=0-shot2025.1249.9
SELFORGBackbone model=Qwen-1.5B2025.1048.97
Vanilla SFTCorrector Sampling=false, Training strategy=SFT, Backbone=LLaDA-Base 8B, Number of shots=02026.0248.17
UM-190kBackbone=Apertus-8B-SFT2025.1148.07
UM-187kBackbone=Apertus-8B-SFT2025.1147.68
Phi-2# Non-Emb Params=2.5B2024.0747.6