Function-level code summarization on Selected repositories Clean code (test)
18.9ROUGE-LAgent4cs
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Agent4csModel=GPT-4o2026.07 | 18.9 | 18.4 | 82.6 | 62.4 | 86.4 | 3.82 | 3.97 | 3.34 | 3.95 | 3.52 | |
| CS-BFModel=GPT-4o2026.07 | 18.1 | 17.9 | 82.2 | 61.5 | 85.9 | 3.79 | 3.94 | 3.28 | 3.95 | 3.53 | |
| Agent4csModel=GPT-4.12026.07 | 17.7 | 18.9 | 82.2 | 67.2 | 88 | 3.75 | 3.99 | 3.37 | 3.96 | 3.61 | |
| HR-CSModel=GPT-4o2026.07 | 17.7 | 17.6 | 82.1 | 60.5 | 87.9 | 3.76 | 3.89 | 3.28 | 3.94 | 3.52 | |
| Agent4csModel=Gemini-2.5-flash2026.07 | 17.2 | 16.4 | 82.2 | 64.7 | 88 | 3.62 | 3.94 | 3.43 | 3.96 | 3.58 | |
| HR-CSModel=GPT-4.12026.07 | 17.2 | 17.9 | 82.2 | 65.6 | 81.2 | 3.64 | 3.97 | 3.29 | 3.94 | 3.6 | |
| CS-BFModel=Gemini-2.5-flash2026.07 | 17.2 | 16.2 | 80.7 | 61.7 | 83.6 | 3.54 | 3.93 | 3.44 | 3.97 | 3.58 | |
| Agent4csModel=GPT-52026.07 | 17.1 | 19.7 | 82.4 | 65 | 77.8 | 3.92 | 4 | 3.92 | 4 | 3.72 | |
| HR-CSModel=Gemini-2.5-flash2026.07 | 17 | 16.6 | 81.2 | 62.2 | 83.2 | 3.53 | 3.92 | 3.41 | 3.96 | 3.51 | |
| Baseline (HR-CS)Model=GPT-52026.07 | 16.6 | 19.3 | 81 | 62.4 | 76.8 | 3.89 | 3.99 | 3.88 | 4 | 3.64 | |
| CS-BFModel=GPT-4.12026.07 | 16.6 | 17.6 | 81.4 | 65.1 | 86.4 | 3.65 | 3.98 | 3.28 | 3.93 | 3.6 | |
| Baseline (CS-BF)Model=GPT-52026.07 | 16.1 | 18.5 | 82.6 | 66.6 | 77.4 | 3.89 | 4 | 3.9 | 4 | 3.6 | |
| Agent4csModel=LLaMA-3.1-8B2026.07 | 15.1 | 15.1 | 81.5 | 62.7 | 77.2 | 3.32 | 3.71 | 2.93 | 3.66 | 3.18 | |
| CS-BFModel=LLaMA-3.1-8B2026.07 | 14.9 | 15.2 | 82.6 | 61.8 | 78.3 | 3.27 | 3.56 | 2.83 | 3.66 | 3.23 | |
| Agent4csModel=Gemma3-4B2026.07 | 14.5 | 14.3 | 81.2 | 60.6 | 78.2 | 3.47 | 3.82 | 2.74 | 3.81 | 1.93 | |
| HR-CSModel=LLaMA-3.1-8B2026.07 | 14.4 | 14.9 | 81.8 | 62.3 | 79.2 | 3.24 | 3.54 | 2.81 | 3.62 | 3.03 | |
| HR-CSModel=Gemma3-4B2026.07 | 14 | 14.7 | 80.3 | 60.1 | 77.2 | 3.38 | 3.79 | 2.99 | 3.75 | 1.9 | |
| CS-BFModel=Gemma3-4B2026.07 | 13.5 | 14.5 | 81 | 59.5 | 77.9 | 3.36 | 3.81 | 3.01 | 3.77 | 2.03 | |
| CS-BFModel=Qwen3-8B2026.07 | 11.8 | 10.9 | 79.8 | 59 | 78.7 | 3.72 | 3.95 | 3.81 | 3.9 | 2.59 | |
| Agent4csModel=Qwen3-8B2026.07 | 11.4 | 11.8 | 81.1 | 58.8 | 79.2 | 3.74 | 3.98 | 3.84 | 3.95 | 1.92 | |
| HR-CSModel=Qwen3-8B2026.07 | 11.3 | 11.2 | 80.3 | 58.6 | 78.3 | 3.69 | 3.94 | 3.78 | 3.92 | 2.68 |