List your company
ResearchFormerly WizWand

Code Generation on HumanEval with PEP 8 compliance n=164 (test)

98.8Pass@1No Retrieval (M0)
87.3690.3393.396.27May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
No Retrieval (M0)Executor=DeepSeek-V4, #Units=02026.0598.89.12
All Rules (M1)Executor=DeepSeek-V4, #Units=1272026.0598.89.32
Std RAG, top-20Executor=DeepSeek-V4, #Units=202026.0598.89.11
Std RAG, top-10Executor=DeepSeek-V4, #Units=102026.0598.49.21
Std RAG, top-15Executor=DeepSeek-V4, #Units=152026.0598.29.2
TAG (Ours)Executor=DeepSeek-V4, #Units=22.02026.0598.19.36
Std RAG, top-5Executor=DeepSeek-V4, #Units=52026.0597.69.13
No Retrieval (M0)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=02026.0593.38.26
All Rules (M1)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=1272026.0590.29.23
TAG (Ours)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=36.12026.0590.29.27
Std RAG, top-20Executor=Qwen3-30B-A3B-Instruct-2507, #Units=202026.05898.85
Std RAG, top-10Executor=Qwen3-30B-A3B-Instruct-2507, #Units=102026.0588.48.8
Std RAG, top-15Executor=Qwen3-30B-A3B-Instruct-2507, #Units=152026.0588.48.7
Std RAG, top-5Executor=Qwen3-30B-A3B-Instruct-2507, #Units=52026.0587.88.63