Repo-level Compilation on CompileAgentBench
96CSRCompileAgent
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CompileAgentBackbone=Claude-3-5-sonnet, Size=-2025.05 | 96 | 5.37 | 22.02 | |
| CompileAgentBackbone=DeepSeek-v2.5, Size=236B2025.05 | 91 | 11.38 | 3.31 | |
| CompileAgentBackbone=GPT-4o, Size=-2025.05 | 89 | 8.38 | 16.53 | |
| CompileAgentBackbone=Qwen2.5-32B-Instruct, Size=32B2025.05 | 80 | 5.25 | 3.16 | |
| Readme-AIBackbone=Claude-3-5-sonnet, Size=-2025.05 | 79 | 127.33 | 55.26 | |
| CompileAgentBackbone=LLama3.1-70B, Size=70B2025.05 | 79 | 7.38 | 2.71 | |
| RAGBackbone=Claude-3-5-sonnet, Size=-2025.05 | 78 | 8.3 | 54.44 | |
| Readme-AIBackbone=GPT-4o, Size=-2025.05 | 72 | 128.8 | 42.94 | |
| RAGBackbone=DeepSeek-v2.5, Size=236B2025.05 | 72 | 11.3 | 36.08 | |
| Readme-AIBackbone=DeepSeek-v2.5, Size=236B2025.05 | 71 | 125.43 | 33.7 | |
| Readme-AIBackbone=Qwen2.5-32B-Instruct, Size=32B2025.05 | 70 | 127.82 | 33.18 | |
| RAGBackbone=GPT-4o, Size=-2025.05 | 67 | 11.12 | 45.78 | |
| CompileAgentBackbone=Gemini-1.5-flash, Size=-2025.05 | 65 | 3.55 | 2.39 | |
| RAGBackbone=Qwen2.5-32B-Instruct, Size=32B2025.05 | 62 | 10.55 | 36.73 | |
| Readme-AIBackbone=LLama3.1-70B, Size=70B2025.05 | 61 | 125.03 | 33.57 | |
| RAGBackbone=LLama3.1-70B, Size=70B2025.05 | 61 | 10.98 | 36.87 | |
| CompileAgentBackbone=Mixtral-8×7B-Instruct, Size=42B2025.05 | 55 | 4.88 | 4.32 | |
| RAGBackbone=Gemini-1.5-flash, Size=-2025.05 | 46 | 9.28 | 35.72 | |
| RAGBackbone=Mixtral-8×7B-Instruct, Size=42B2025.05 | 45 | 10.82 | 36.49 | |
| Readme-AIBackbone=Gemini-1.5-flash, Size=-2025.05 | 41 | 123.68 | 32.37 | |
| Readme-AIBackbone=Mixtral-8×7B-Instruct, Size=42B2025.05 | 38 | 124.6 | 33.12 | |
| Oss-Fuzz-GenNote=Operates without relying on LLMs2025.05 | 25 | 53.01 | — |