NL2Code Search on CodeSearchNet (test)
86.85Recall (Python)CODEBERT (MLM+RTD)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CODEBERT (MLM+RTD)INIT=R, Learning Objective=MLM+RTD, Initialization=RoBERTa2020.02 | 86.85 | 74.84 | 70.59 | 70.62 | 84 | 69.26 | 76.03 | |
| CODEBERT (MLM)INIT=R, Learning Objective=MLM, Initialization=RoBERTa2020.02 | 86.47 | 74.76 | 69.97 | 68.93 | 83.83 | 68.98 | 75.49 | |
| PT w/ CODE ONLYINIT=R, Pre-training Data=Codes only, Initialization=RoBERTa2020.02 | 84.38 | 72.13 | 64.02 | 67.06 | 81.91 | 66.12 | 72.6 | |
| CODEBERT (RTD)INIT=R, Learning Objective=RTD, Initialization=RoBERTa2020.02 | 82.63 | 71.5 | 65.12 | 67.74 | 82.85 | 64.14 | 72.33 | |
| CODEBERT (MLM)INIT=S, Learning Objective=MLM, Initialization=From scratch2020.02 | 82.61 | 71.42 | 60.29 | 65.56 | 83.04 | 56.95 | 69.98 | |
| ROBERTAPre-training=Natural Language2020.02 | 80.87 | 66.59 | 60.6 | 65.76 | 82.04 | 62.45 | 69.72 | |
| PT w/ CODE ONLYINIT=S, Pre-training Data=Codes only, Initialization=From scratch2020.02 | 78.55 | 65.67 | 55.57 | 61.72 | 79.29 | 57.12 | 66.32 | |
| OASISModel Type=Open-source, Setting=zero-shot2025.03 | 73.69 | 73.97 | 69.8 | 63.84 | 88.21 | 75.47 | 74.16 | |
| Text-Embedding-3-LargeModel Type=Closed-source, Setting=zero-shot2025.03 | 70.84 | 72.92 | 68.13 | 59.59 | 87.64 | 75.25 | 72.4 | |
| CodeSage-largeModel Type=Open-source, Setting=zero-shot2025.03 | 70.77 | 70.21 | 69.5 | 61.33 | 83.71 | 71.92 | 71.24 | |
| SELFATTType=Joint Embedding2020.02 | 69.22 | 58.66 | 45.06 | 60.11 | 68.09 | 36.51 | 56.28 | |
| OpenAI-ada-002Model Type=Closed-source, Setting=zero-shot2025.03 | 68.02 | 71.49 | 67.5 | 60.62 | 85.63 | 74.72 | 71.33 | |
| NBOWType=Joint Embedding2020.02 | 58.09 | 51.4 | 46.07 | 48.35 | 64.09 | 42.85 | 51.81 | |
| CNNType=Joint Embedding2020.02 | 57.08 | 52.7 | 35.23 | 52.94 | 62.74 | 24.5 | 47.53 | |
| UnixCoderModel Type=Open-source, Setting=zero-shot2025.03 | 42.17 | 43.92 | 40.46 | 35.21 | 61.39 | 55.22 | 46.39 | |
| BIRNNType=Joint Embedding2020.02 | 32.13 | 28.65 | 15.3 | 25.12 | 45.24 | 8.35 | 25.8 | |
| GraphCodeBERTModel Type=Open-source, Setting=zero-shot2025.03 | 10.37 | 8.59 | 7.29 | 8.07 | 12.47 | 20.79 | 11.26 | |
| CodeBERTModel Type=Open-source, Setting=zero-shot2025.03 | 0.05 | 0.03 | 0.04 | 0.01 | 0.02 | 0.14 | 0.1 |