Automated Exploit Generation on Penetration Testing Benchmark (test)
90ASR (SI)AutoEG
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AutoEGLLM Backbone=DeepSeek-V32026.04 | 90 | 78 | 93 | 94 | 57 | 83 | 83 | 88 | |
| AutoEGLLM Backbone=GPT-4o2026.04 | 80 | 82 | 75 | 94 | 80 | 76 | 67 | 80 | |
| AutoEGLLM Backbone=Qwen-Plus2026.04 | 75 | 81 | 93 | 88 | 94 | 97 | 92 | 87 | |
| AutoEGLLM Backbone=Claude-3.72026.04 | 70 | 65 | 80 | 58 | 60 | 76 | 75 | 72 | |
| VulnBotLLM Backbone=Claude-3.72026.04 | 54 | 32 | 43 | 23 | 23 | 58 | 25 | 37 | |
| VulnBotLLM Backbone=DeepSeek-V32026.04 | 50 | 41 | 38 | 26 | 19 | 51 | 23 | 35 | |
| VulnBotLLM Backbone=GPT-4o2026.04 | 48 | 23 | 38 | 19 | 20 | 52 | 22 | 32 | |
| PentestAgentLLM Backbone=GPT-4o2026.04 | 46 | 23 | 7 | 20 | 25 | 0 | 0 | 21 | |
| VulnBotLLM Backbone=Qwen-Plus2026.04 | 44 | 20 | 32 | 20 | 14 | 42 | 20 | 27 | |
| PentestGPT-AutoLLM Backbone=GPT-4o2026.04 | 38 | 11 | 25 | 0 | 0 | 25 | 0 | 20 | |
| PentestGPT-AutoLLM Backbone=Qwen-Plus2026.04 | 35 | 19 | 26 | 20 | 0 | 25 | 17 | 20 | |
| PentestGPT-AutoLLM Backbone=Claude-3.72026.04 | 34 | 45 | 28 | 30 | 34 | 43 | 50 | 38 | |
| PentestAgentLLM Backbone=Qwen-Plus2026.04 | 29 | 16 | 30 | 20 | 50 | 13 | 0 | 23 | |
| PentestAgentLLM Backbone=Claude-3.72026.04 | 29 | 10 | 7 | 0 | 25 | 0 | 0 | 13 | |
| PentestAgentLLM Backbone=DeepSeek-V32026.04 | 25 | 26 | 26 | 20 | 50 | 0 | 50 | 23 | |
| DireLLMLLM Backbone=GPT-4o2026.04 | 23 | 11 | 7 | 10 | 50 | 19 | 0 | 15 | |
| PentestGPT-AutoLLM Backbone=DeepSeek-V32026.04 | 23 | 39 | 35 | 40 | 34 | 38 | 32 | 34 | |
| PentestGPT-Auto-3LLM Backbone=DeepSeek-V32026.04 | 23 | 9 | 15 | 10 | 16 | 13 | 13 | 14 | |
| DireLLMLLM Backbone=Qwen-Plus2026.04 | 21 | 15 | 13 | 20 | 13 | 19 | 13 | 16 | |
| PentestGPT-Auto-3LLM Backbone=Qwen-Plus2026.04 | 17 | 7 | 16 | 0 | 11 | 8 | 17 | 11 | |
| PentestGPT-Auto-3LLM Backbone=Claude-3.72026.04 | 17 | 5 | 22 | 20 | 17 | 7 | 0 | 12 | |
| PentestGPT-Auto-3LLM Backbone=GPT-4o2026.04 | 13 | 7 | 10 | 0 | 0 | 0 | 0 | 8 | |
| DireLLMLLM Backbone=Claude-3.72026.04 | 13 | 10 | 9 | 10 | 25 | 6 | 0 | 11 | |
| DireLLMLLM Backbone=DeepSeek-V32026.04 | 10 | 3 | 7 | 10 | 0 | 6 | 13 | 7 | |
| VulnBot-3LLM Backbone=Claude-3.72026.04 | 10 | 5 | 8 | 0 | 0 | 11 | 0 | 5 | |
| VulnBot-3LLM Backbone=GPT-4o2026.04 | 9 | 4 | 7 | 0 | 0 | 10 | 0 | 4 | |
| VulnBot-3LLM Backbone=DeepSeek-V32026.04 | 9 | 0 | 7 | 0 | 0 | 11 | 0 | 4 | |
| VulnBot-3LLM Backbone=Qwen-Plus2026.04 | 7 | 0 | 5 | 0 | 0 | 11 | 1 | 3 |