List your company
ResearchFormerly WizWand

Code Generation on MBPP standard (test)

94.2Pass@1QualityFlow
35.4450.69565.9581.205May 28, 2024Jul 6, 2024Aug 15, 2024Sep 23, 2024Nov 2, 2024Dec 11, 2024Jan 20, 2025
Updated 7mo ago

Evaluation results

MethodLinks
QualityFlowLLM Backbone=Claude Sonnet-3.5-v22025.0194.2——4.8
DeepSeek-Coder-V2-Instruct2025.0189.4———
LEDEX RLBackbone=CodeLlama-13B, Stage=Refine2024.0561.98—79.95—
LEDEX SFTBackbone=CodeLlama-13B, Stage=Refine2024.0561.78—79.96—
LEDEX RLBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.0561.63—80.27—
LEDEX SFTBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.0561.59—80.21—
LEDEX RLBackbone=StarCoder-15B, Stage=Expl. + Refine2024.0558.19—77.96—
LEDEX SFTBackbone=CodeLlama-7B, Stage=Refine2024.0558.07—77.34—
LEDEX RLBackbone=StarCoder-15B, Stage=Refine2024.0558—78.12—
LEDEX SFTBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.0557.98—77.92—
LEDEX RLBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.0557.92—78.97—
LEDEX RLBackbone=CodeLlama-7B, Stage=Refine2024.0557.35—78.12—
LEDEX SFTBackbone=StarCoder-15B, Stage=Expl. + Refine2024.0557.11—76.7—
LEDEX SFTBackbone=StarCoder-15B, Stage=Refine2024.0556.66—75.78—
LEDEX SFTBackbone=CodeLlama-13B, Stage=Init2024.0552.43—73.66—
LEDEX RLBackbone=CodeLlama-13B, Stage=Init2024.0551.19—73.16—
PromptBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.0549.97—76.39—
PromptBackbone=CodeLlama-13B, Stage=Refine2024.0549.68—75.85—
LEDEX SFTBackbone=CodeLlama-7B, Stage=Init2024.0548.87—70.89—
LEDEX RLBackbone=StarCoder-15B, Stage=Init2024.0548.62—70.68—
LEDEX SFTBackbone=StarCoder-15B, Stage=Init2024.0547.41—70.27—
LEDEX RLBackbone=CodeLlama-7B, Stage=Init2024.0546.54—71.54—
PromptBackbone=CodeLlama-7B, Stage=Refine2024.0543.98—71.94—
PromptBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.0543.42—72.09—
PromptBackbone=CodeLlama-13B, Stage=Init2024.0542.88—70.85—
PromptBackbone=StarCoder-15B, Stage=Refine2024.0540.64—71.04—
PromptBackbone=StarCoder-15B, Stage=Expl. + Refine2024.0540.37—71.6—
PromptBackbone=CodeLlama-7B, Stage=Init2024.0538.21—67.24—
PromptBackbone=StarCoder-15B, Stage=Init2024.0537.7—69.6—
Source Model baselineModel=Llama-2-7b, PEFT=LoRA, Protocol=Zero-shot2024.05—27.2——
Source Model baselineModel=Gemma-2b, PEFT=LoRA, Protocol=Zero-shot2024.05—41.1——
Target Model baselineModel=Llama-2-13b, PEFT=None, Protocol=Zero-shot2024.05—37.1——
Target Model baselineModel=Gemma-7b, PEFT=None, Protocol=Zero-shot2024.05—37.9——
Trans-LoRASource Model=Llama-2-7b, Target Model=Llama-2-13b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05—39.7——
Trans-LoRASource Model=Gemma-2b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05—50——
Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05—48.7——
Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05—48.7——
Trans-LoRASource Model=Llama2-7B, Configuration=Source LoRA2026.03—27.2——
Trans-LoRATarget Model=Llama2-13B, Configuration=Target (no LoRA)2026.03—37.1——
Trans-LoRASource Model=Llama2-7B, Target Model=Llama2-13B, Configuration=Target LoRA2026.03—39.7——
Trans-LoRASource Model=Gemma-2B, Configuration=Source LoRA2026.03—41.1——
Trans-LoRATarget Model=Gemma-7B, Configuration=Target (no LoRA)2026.03—37.9——
Trans-LoRASource Model=Gemma-2B, Target Model=Gemma-7B, Configuration=Target LoRA2026.03—50——
TuneShift-KDSource Model=Llama2-7B, Configuration=Source LoRA (Reproduced)2026.03—28.2——
TuneShift-KDTarget Model=Llama2-13B, Configuration=Target (no LoRA) (Reproduced)2026.03—36.9——
TuneShift-KDSource Model=Llama2-7B, Target Model=Llama2-13B, Configuration=Target LoRA2026.03—40.2——
TuneShift-KDSource Model=Gemma-2B, Configuration=Source LoRA (Reproduced)2026.03—40.9——
TuneShift-KDTarget Model=Gemma-7B, Configuration=Target (no LoRA) (Reproduced)2026.03—37.8——
TuneShift-KDSource Model=Gemma-2B, Target Model=Gemma-7B, Configuration=Target LoRA2026.03—51.3——