Code Generation on HumanEval with PEP 8 compliance n=164 (test)
98.8Pass@1No Retrieval (M0)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| No Retrieval (M0)Executor=DeepSeek-V4, #Units=02026.05 | 98.8 | 9.12 | |
| All Rules (M1)Executor=DeepSeek-V4, #Units=1272026.05 | 98.8 | 9.32 | |
| Std RAG, top-20Executor=DeepSeek-V4, #Units=202026.05 | 98.8 | 9.11 | |
| Std RAG, top-10Executor=DeepSeek-V4, #Units=102026.05 | 98.4 | 9.21 | |
| Std RAG, top-15Executor=DeepSeek-V4, #Units=152026.05 | 98.2 | 9.2 | |
| TAG (Ours)Executor=DeepSeek-V4, #Units=22.02026.05 | 98.1 | 9.36 | |
| Std RAG, top-5Executor=DeepSeek-V4, #Units=52026.05 | 97.6 | 9.13 | |
| No Retrieval (M0)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=02026.05 | 93.3 | 8.26 | |
| All Rules (M1)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=1272026.05 | 90.2 | 9.23 | |
| TAG (Ours)Executor=Qwen3-30B-A3B-Instruct-2507, #Units=36.12026.05 | 90.2 | 9.27 | |
| Std RAG, top-20Executor=Qwen3-30B-A3B-Instruct-2507, #Units=202026.05 | 89 | 8.85 | |
| Std RAG, top-10Executor=Qwen3-30B-A3B-Instruct-2507, #Units=102026.05 | 88.4 | 8.8 | |
| Std RAG, top-15Executor=Qwen3-30B-A3B-Instruct-2507, #Units=152026.05 | 88.4 | 8.7 | |
| Std RAG, top-5Executor=Qwen3-30B-A3B-Instruct-2507, #Units=52026.05 | 87.8 | 8.63 |