Programming by Example on LLM-PBE Attacked
35AccuracyDeepSeek-Coder-V2 16B
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek-Coder-V2 16BBackend=Ollama2026.07 | 35 | |
| Qwen2.5-Coder 1.5BBackend=Ollama2026.07 | 25 | |
| Qwen3.7-plus APITemperature=0, Token cap=512, Request type=stateless2026.07 | 15 | |
| Qwen2.5-Coder 7BBackend=Ollama2026.07 | 10 | |
| DeepSeek-Coder-V2 16B 8kBackend=Ollama2026.07 | 5 | |
| Qwen2.5-Coder 14BBackend=Ollama2026.07 | 0 | |
| Qwen2.5-Coder 32B CPUBackend=Ollama2026.07 | 0 | |
| Kimi-k2.7-code APITemperature=0, Token cap=512, Request type=stateless2026.07 | 0 | |
| GLM-5.2 APITemperature=0, Token cap=512, Request type=stateless2026.07 | 0 | |
| gpt-5.5 APITemperature=0, Token cap=512, Request type=stateless2026.07 | 0 | |
| claude-opus-4-8 APITemperature=0, Token cap=512, Request type=stateless2026.07 | 0 | |
| gemini-3.1-pro-high APITemperature=0, Token cap=512, Request type=stateless2026.07 | 0 |