List your company
ResearchFormerly WizWand

Code Generation on MBPP+ (Pass@1, AVG)

75.4Pass@1CODEBLOCK
52.83258.69164.5570.409Apr 14, 2026Apr 23, 2026May 3, 2026May 12, 2026May 22, 2026May 31, 2026Jun 10, 2026
Updated 4mo ago

Evaluation results

MethodLinks
CODEBLOCKBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.0675.4—
DS2Base model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=4.62026.0674.1—
CLAMBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=3.52026.0674.1—
TOKEN CLEANINGBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.72026.0673.3—
RANDOM SELECTIONBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=1.92026.0673.1—
MTModel=Qwen2.5-Coder-14B-Inst2026.0472.884.07
DS2Base model=Seed-Coder-8B, Eff. Tokens (%)=4.62026.0672.466
BASEBase model=Qwen2.5-Coder-7B-Instruct2026.0671.6—
FULL TOKENSBase model=Qwen2.5-Coder-7B-Instruct, Eff. Tokens (%)=100.02026.0671.4—
VCRDTeacher=Qwen2.5-Coder-14B-Inst, Student=Qwen2.5-Coder-7B-Inst2026.0471.281.75
CLAMBase model=Seed-Coder-8B, Eff. Tokens (%)=3.62026.0671.164.9
DistillLM-2Teacher=Qwen2.5-Coder-14B-Inst, Student=Qwen2.5-Coder-7B-Inst2026.0470.981.37
FULL TOKENSBase model=Seed-Coder-8B, Eff. Tokens (%)=100.02026.0670.865.1
RANDOM SELECTIONBase model=Seed-Coder-8B, Eff. Tokens (%)=1.92026.0670.663.9
DistilLLMTeacher=Qwen2.5-Coder-14B-Inst, Student=Qwen2.5-Coder-7B-Inst2026.0470.480.75
CODEBLOCKBase model=Seed-Coder-8B, Eff. Tokens (%)=1.92026.0670.465.5
TOKEN CLEANINGBase model=Seed-Coder-8B, Eff. Tokens (%)=1.72026.0669.862.8
FULL TOKENSBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=100.02026.0669.6—
BASEBase model=Seed-Coder-8B2026.066962.5
FULL TOKENSBase model=OpenCoder-8B-Base, Eff. Tokens (%)=100.02026.0668.754.5
BASEBase model=OpenCoder-8B-Base2026.0668.554.4
RANDOM SELECTIONBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.92026.066854.6
DS2Base model=OpenCoder-8B-Base, Eff. Tokens (%)=4.62026.066854.6
DS2Base model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=4.62026.0668—
CODEBLOCKBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.0668—
CLAMBase model=OpenCoder-8B-Base, Eff. Tokens (%)=3.52026.0667.754.6
CLAMBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=3.52026.0667.7—
TOKEN CLEANINGBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.72026.0667.5—
TOKEN CLEANINGBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.72026.0666.951.3
RANDOM SELECTIONBase model=Qwen2.5-Coder-3B-Instruct, Eff. Tokens (%)=1.92026.0666.7—
SFTBackbone=Qwen2.5-Coder-3B2026.0666.67—
ASFTBackbone=Qwen2.5-Coder-3B2026.0666.4—
CODEBLOCKBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.92026.0666.457.1
CLAMBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=3.52026.0666.151.8
IDFTBackbone=Qwen2.5-Coder-3B2026.0665.87—
DFTBackbone=Qwen2.5-Coder-3B2026.0665.34—
TALRBackbone=Qwen2.5-Coder-3B2026.0665.34—
EAFTBackbone=Qwen2.5-Coder-3B2026.0664.81—
RANDOM SELECTIONBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.92026.0664.653.6
DS2Base model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=4.62026.0664.653.6
PriFT-massBackbone=Qwen2.5-Coder-3B2026.0664.29—
CODEBLOCKBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.92026.0663.554.6
OriginalBackbone=Qwen2.5-Coder-3B2026.0663.49—
BASEBase model=Qwen2.5-Coder-3B-Instruct2026.0662.7—
PriFT-probBackbone=Qwen2.5-Coder-3B2026.0662.43—
FULL TOKENSBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=100.02026.0662.452.7
BASEBase model=Qwen2.5-Coder-1.5B-Instruct2026.0659.849.4
TOKEN CLEANINGBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.82026.0653.749.1
ARES-RLBackbone=Qwen3-32B2026.05—63.16
ARES-SFTBackbone=Qwen3-32B2026.05—61.4
AWQModel Scale=Llama-3.1-3B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—56.08
AWQModel Scale=Llama-3.1-8B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—60.85
CPTBackbone=Qwen3-32B2026.05—59.4
E-PMQModel Scale=Llama-3.1-3B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—57.14
E-PMQModel Scale=Llama-3.1-8B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—61.38
GPTQModel Scale=Llama-3.1-3B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—55.56
GPTQModel Scale=Llama-3.1-8B, Quantization Bit-width=4-bit, Merging Algorithm=Task Arithmetic2026.05—57.94
NaturalReasoningBackbone=Qwen3-32B2026.05—61.15
Task ArithmeticModel Scale=Llama-3.1-3B, Quantization Bit-width=Full-precision, Merging Algorithm=Task Arithmetic2026.05—57.94
Task ArithmeticModel Scale=Llama-3.1-8B, Quantization Bit-width=Full-precision, Merging Algorithm=Task Arithmetic2026.05—61.38
WebscaleBackbone=Qwen3-32B2026.05—61.4