Code Generation on HumanEval (pass@1, pass@5)
53.05Pass@1CAA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CAABackbone=Llama-3.2-3B-Instruct, Synthetic Data Strategy=GUIDEDSAMPLING - Concepts and Answer (CAA)2025.10 | 53.05 | 59.21 | |
| RSBackbone=Llama-3.2-3B-Instruct, Synthetic Data Strategy=Repeated Sampling2025.10 | 52.13 | 55.78 | |
| STaRBackbone=Llama-3.2-3B-Instruct, Synthetic Data Strategy=Self-Taught Reasoner2025.10 | 52.13 | 57.35 | |
| FABackbone=Llama-3.2-3B-Instruct, Synthetic Data Strategy=GUIDEDSAMPLING - Final Answer (FA)2025.10 | 48.17 | 55.95 | |
| No WatermarkBackbone Model=Starcoder2025.05 | 43 | — | |
| UnbiasedBackbone Model=Starcoder2025.05 | 36 | — | |
| DiPmarkBackbone Model=Starcoder2025.05 | 36 | — | |
| ToTBackbone=Llama-3.2-3B-Instruct, Synthetic Data Strategy=Tree-of-Thought2025.10 | 35.73 | 49.51 | |
| E2E-LLM-WatermarkBackbone Model=Starcoder2025.05 | 34 | — | |
| UnigramBackbone Model=Starcoder2025.05 | 33 | — | |
| Base ModelBackbone=Llama-3.2-3B-Instruct2025.10 | 27.44 | 39.02 | |
| KGWBackbone Model=Starcoder2025.05 | 22 | — |