Code Summarization on CodeXGLUE
20.83Java ScoreCodeT5+
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CodeT5+Number of Parameters=770M2023.05 | 20.83 | 20.47 | 17.93 | 20.15 | 15.63 | 19.64 | 26.39 | |
| CodeT5+Number of Parameters=220M2023.05 | 20.53 | 20.16 | 16.27 | 19.81 | 15.51 | 19.6 | 26.78 | |
| UniXcoderNumber of Parameters=125M2023.05 | 20.31 | 19.13 | 15.85 | 19.3 | 14.87 | 19.07 | 26.54 | |
| CodeT5Number of Parameters=220M2023.05 | 20.31 | 20.01 | 16.16 | 19.55 | 15.24 | 19.56 | 26.03 | |
| PolyglotCODEBERTtraining=multi-lingual fine-tuning2021.12 | 20.11 | 18.71 | 15.8 | 19.06 | 14.75 | 18.77 | 26.23 | |
| PolyglotGRAPHCODEBERTtraining=multi-lingual fine-tuning2021.12 | 19.91 | 18.9 | 15.79 | 19.1 | 14.95 | 18.92 | 26.15 | |
| CodeGen-multiNumber of Parameters=350M2023.05 | 19.41 | 18.31 | 16.54 | 18.37 | 13.48 | 18.09 | 24.41 | |
| ProphetNet-Xtraining=monolingual fine-tuning2021.12 | 19.39 | 17.87 | 16.6 | 18.54 | 14.37 | 18.43 | 24.57 | |
| GRAPHCODEBERTtraining=monolingual fine-tuning2021.12 | 19.22 | 18.02 | 14.79 | 18.08 | 12.62 | 18.4 | 25.45 | |
| PLBARTtraining=monolingual fine-tuning2021.12 | 18.45 | 19.3 | 15.56 | 18.32 | 14.11 | 18.91 | 23.58 | |
| PLBARTNumber of Parameters=140M2023.05 | 18.45 | 19.3 | 15.56 | 18.32 | 14.11 | 18.91 | 23.58 | |
| GALLaBackbone=Qwen2.5-Coder 14B, Setting=G2C2024.09 | 18.1 | 19.9 | 15.7 | — | — | — | — | |
| Qwen2.5-Coder 14BSetting=Baseline2024.09 | 18 | 20.2 | 14.5 | — | — | — | — | |
| GALLaBackbone=Qwen2.5-Coder 14B, Setting=G2C + GraphQA2024.09 | 17.9 | 20.5 | 15.9 | — | — | — | — | |
| CODEBERTtraining=monolingual fine-tuning2021.12 | 17.65 | 19.06 | 14.9 | 17.83 | 12.16 | 18.07 | 25.16 | |
| CodeBERTNumber of Parameters=125M2023.05 | 17.65 | 19.06 | 14.9 | 17.83 | 12.16 | 18.07 | 25.16 | |
| RoBERTa2021.12 | 16.47 | 18.14 | 11.9 | 16.57 | 11.17 | 17.72 | 24.02 | |
| ROBERTaNumber of Parameters=125M2023.05 | 16.47 | 18.14 | 11.9 | 16.57 | 11.17 | 17.72 | 24.02 | |
| Transformer2021.12 | 16.26 | 15.81 | 11.59 | 15.56 | 11.18 | 16.38 | 22.12 | |
| GALLaBackbone=Phi-1 1.3B, Setting=G2C + GraphQA2024.09 | 15.2 | 11.7 | 13.6 | — | — | — | — | |
| GALLaBackbone=Phi-1 1.3B, Setting=G2C2024.09 | 15.1 | 11.1 | 13.4 | — | — | — | — | |
| Seq2Seq2021.12 | 15.09 | 15.93 | 10.21 | 14.32 | 9.64 | 13.98 | 21.08 | |
| GALLaBackbone=Qwen2.5-Coder 1.5B, Setting=G2C + GraphQA2024.09 | 15 | 11.3 | 14.3 | — | — | — | — | |
| GALLaBackbone=Qwen2.5-Coder 1.5B, Setting=G2C2024.09 | 14.7 | 11.5 | 13.4 | — | — | — | — | |
| Phi-1 1.3BSetting=Baseline2024.09 | 14.6 | 11.3 | 13.8 | — | — | — | — | |
| GALLaBackbone=LLAMA3 8B, Setting=G2C + GraphQA2024.09 | 14.5 | 11.7 | 12.9 | — | — | — | — | |
| Qwen2.5-Coder 1.5BSetting=Baseline2024.09 | 14.4 | 10.8 | 13.6 | — | — | — | — | |
| LLAMA3 8BSetting=Baseline2024.09 | 14.2 | 11.2 | 12.3 | — | — | — | — | |
| GALLaBackbone=CodeGen 350M, Setting=G2C + GraphQA2024.09 | 14 | 9.5 | 12.7 | — | — | — | — | |
| GALLaBackbone=LLAMA3 8B, Setting=G2C2024.09 | 14 | 11.7 | 11.8 | — | — | — | — | |
| GALLaBackbone=CodeGen 350M, Setting=G2C2024.09 | 13.9 | 9.5 | 12.8 | — | — | — | — | |
| CodeGen 350MSetting=Baseline2024.09 | 13.6 | 8.8 | 12.5 | — | — | — | — | |
| GALLaBackbone=StarCoder 1B, Setting=G2C + GraphQA2024.09 | 9.5 | 4.5 | 7.4 | — | — | — | — | |
| GALLaBackbone=StarCoder 1B, Setting=G2C2024.09 | 9.4 | 4.3 | 6.8 | — | — | — | — | |
| StarCoder 1BSetting=Baseline2024.09 | 7.8 | 3.2 | 5.1 | — | — | — | — | |
| GALLaBackbone=LLAMA2 7B, Setting=G2C2024.09 | 2.4 | 1.7 | 1.9 | — | — | — | — | |
| GALLaBackbone=LLAMA2 7B, Setting=G2C + GraphQA2024.09 | 2.2 | 1.6 | 2 | — | — | — | — | |
| LLAMA2 7BSetting=Baseline2024.09 | 2 | 1.5 | 1.8 | — | — | — | — |