List your company
ResearchFormerly WizWand

Code Generation on DS-1000

69.9AccuracyEG-CFG
18.00431.47744.9558.423Jun 12, 2025Aug 3, 2025Sep 24, 2025Nov 16, 2025Jan 7, 2026Feb 28, 2026Apr 22, 2026
Updated 6mo ago

Evaluation results

MethodLinks
EG-CFGModel=DeepSeek-V3-03242025.0669.950.73
CONLINEModel=GPT-42025.0668—
TextGradActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_TG → πA2026.0463—
DSPyActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_DSPy → πA2026.0462—
SGT (DPO Iteration 4)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_4 → πA2026.0462—
SGT (DPO Iteration 5)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_5 → πA2026.0461.5—
SGT (DPO Iteration 1)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_1 → πA2026.0461—
Baseline LLMModel=GPT-42025.0660.2—
SFT SupplementActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_SFT → πA2026.0460—
Baseline LLMModel=GPT-4o2025.0659.9—
TextGradActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_TG → πA2026.0459.5—
SGT (DPO Iteration 3)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_3 → πA2026.0459—
Prompted SupplementActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_prompt → πA2026.0458.5—
SGT (DPO Iteration 2)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_2 → πA2026.0458.5—
Inference-time scaling (ITS)Actor Model=v3.5-sonnet-v2, Supplement Model=None, Supplement Generation Method=∅ → πA_ITS2026.0458—
DSPyActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_DSPy → πA2026.0457.5—
SelfEvolveModel=GPT-3.5 Turbo2025.0657.1—
SGT (DPO Iteration 5)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_5 → πA2026.0457—
Baseline ActorActor Model=v3.5-sonnet-v2, Supplement Model=None, Supplement Generation Method=∅ → πA2026.0456.5—
SGT (DPO Iteration 3)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_3 → πA2026.0456.5—
Inference-time scaling (ITS)Actor Model=gpt-oss-120b, Supplement Model=None, Supplement Generation Method=∅ → πA_ITS2026.0455—
SFT SupplementActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_SFT → πA2026.0454.5—
SGT (DPO Iteration 1)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_1 → πA2026.0454.5—
Baseline LLMModel=Claude 3.5 Sonnet2025.0654.3—
Baseline ActorActor Model=gpt-oss-120b, Supplement Model=None, Supplement Generation Method=∅ → πA2026.0454—
Baseline LLMModel=DeepSeek-Coder-V2-SFT2025.0653.2—
Self DebuggingModel=GPT-3.5 Turbo2025.0653—
Baseline LLMModel=Qwen2-72B-Instruct2025.0652.8—
SGT (DPO Iteration 4)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_4 → πA2026.0452—
SGT (DPO Iteration 2)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_2 → πA2026.0451—
Prompted SupplementActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_prompt → πA2026.0449.5—
DocPromptingModel=GPT-3.5 Turbo2025.0645.5—
Baseline LLMModel=DeepSeek-V3-03242025.0638.90
Supplement Solver OnlyActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=∅ → πS_solve2026.0420—
Supplement Solver OnlyActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=∅ → πS_solve2026.0420—