Code Generation on DS-1000
69.9AccuracyEG-CFG
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| EG-CFGModel=DeepSeek-V3-03242025.06 | 69.9 | 50.73 | |
| CONLINEModel=GPT-42025.06 | 68 | — | |
| TextGradActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_TG → πA2026.04 | 63 | — | |
| DSPyActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_DSPy → πA2026.04 | 62 | — | |
| SGT (DPO Iteration 4)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_4 → πA2026.04 | 62 | — | |
| SGT (DPO Iteration 5)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_5 → πA2026.04 | 61.5 | — | |
| SGT (DPO Iteration 1)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_1 → πA2026.04 | 61 | — | |
| Baseline LLMModel=GPT-42025.06 | 60.2 | — | |
| SFT SupplementActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_SFT → πA2026.04 | 60 | — | |
| Baseline LLMModel=GPT-4o2025.06 | 59.9 | — | |
| TextGradActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_TG → πA2026.04 | 59.5 | — | |
| SGT (DPO Iteration 3)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_3 → πA2026.04 | 59 | — | |
| Prompted SupplementActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_prompt → πA2026.04 | 58.5 | — | |
| SGT (DPO Iteration 2)Actor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_2 → πA2026.04 | 58.5 | — | |
| Inference-time scaling (ITS)Actor Model=v3.5-sonnet-v2, Supplement Model=None, Supplement Generation Method=∅ → πA_ITS2026.04 | 58 | — | |
| DSPyActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_DSPy → πA2026.04 | 57.5 | — | |
| SelfEvolveModel=GPT-3.5 Turbo2025.06 | 57.1 | — | |
| SGT (DPO Iteration 5)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_5 → πA2026.04 | 57 | — | |
| Baseline ActorActor Model=v3.5-sonnet-v2, Supplement Model=None, Supplement Generation Method=∅ → πA2026.04 | 56.5 | — | |
| SGT (DPO Iteration 3)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_3 → πA2026.04 | 56.5 | — | |
| Inference-time scaling (ITS)Actor Model=gpt-oss-120b, Supplement Model=None, Supplement Generation Method=∅ → πA_ITS2026.04 | 55 | — | |
| SFT SupplementActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_SFT → πA2026.04 | 54.5 | — | |
| SGT (DPO Iteration 1)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_1 → πA2026.04 | 54.5 | — | |
| Baseline LLMModel=Claude 3.5 Sonnet2025.06 | 54.3 | — | |
| Baseline ActorActor Model=gpt-oss-120b, Supplement Model=None, Supplement Generation Method=∅ → πA2026.04 | 54 | — | |
| Baseline LLMModel=DeepSeek-Coder-V2-SFT2025.06 | 53.2 | — | |
| Self DebuggingModel=GPT-3.5 Turbo2025.06 | 53 | — | |
| Baseline LLMModel=Qwen2-72B-Instruct2025.06 | 52.8 | — | |
| SGT (DPO Iteration 4)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_4 → πA2026.04 | 52 | — | |
| SGT (DPO Iteration 2)Actor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_2 → πA2026.04 | 51 | — | |
| Prompted SupplementActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=πS_prompt → πA2026.04 | 49.5 | — | |
| DocPromptingModel=GPT-3.5 Turbo2025.06 | 45.5 | — | |
| Baseline LLMModel=DeepSeek-V3-03242025.06 | 38.9 | 0 | |
| Supplement Solver OnlyActor Model=v3.5-sonnet-v2, Supplement Model=Qwen3-1.7B, Supplement Generation Method=∅ → πS_solve2026.04 | 20 | — | |
| Supplement Solver OnlyActor Model=gpt-oss-120b, Supplement Model=Qwen3-1.7B, Supplement Generation Method=∅ → πS_solve2026.04 | 20 | — |