Code Retrieval on CodeSearch (Language Breakdown)
47.5MRRSANTA
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| SANTAEvaluation Protocol=Fine-Tuning, Training Strategy=Hard Negative2023.05 | 47.5 | — | — | — | — | — | — | — | |
| SANTAEvaluation Protocol=Fine-Tuning, Training Strategy=Inbatch2023.05 | 47.3 | — | — | — | — | — | — | — | |
| CodeRetriever (AR2)Evaluation Protocol=Fine-Tuning2023.05 | 46.9 | — | — | — | — | — | — | — | |
| SANTAEvaluation Protocol=Zero-Shot2023.05 | 46.1 | — | — | — | — | — | — | — | |
| CodeRetrieverEvaluation Protocol=Fine-Tuning2023.05 | 43 | — | — | — | — | — | — | — | |
| CodeT5Evaluation Protocol=Fine-Tuning2023.05 | 39.3 | — | — | — | — | — | — | — | |
| CodeRetrieverEvaluation Protocol=Zero-Shot2023.05 | 34.7 | — | — | — | — | — | — | — | |
| CodeBERTEvaluation Protocol=Fine-Tuning2023.05 | 27.2 | — | — | — | — | — | — | — | |
| T5Evaluation Protocol=Fine-Tuning2023.05 | 23.8 | — | — | — | — | — | — | — | |
| RoBERTaEvaluation Protocol=Fine-Tuning2023.05 | 18.3 | — | — | — | — | — | — | — | |
| BERTEvaluation Protocol=Fine-Tuning2023.05 | 16.7 | — | — | — | — | — | — | — | |
| BERTEvaluation Protocol=Zero-Shot2023.05 | 0.2 | — | — | — | — | — | — | — | |
| RoBERTaEvaluation Protocol=Zero-Shot2023.05 | 0.03 | — | — | — | — | — | — | — | |
| CodeBERTEvaluation Protocol=Zero-Shot2023.05 | 0.03 | — | — | — | — | — | — | — | |
| T5Evaluation Protocol=Zero-Shot2023.05 | 0.03 | — | — | — | — | — | — | — | |
| CodeT5Evaluation Protocol=Zero-Shot2023.05 | 0.03 | — | — | — | — | — | — | — | |
| CodeBERTEvaluation Protocol=Fine-Tuning2023.05 | — | 67.9 | 62 | 88.2 | 67.2 | 67.6 | 62.8 | 69.3 | |
| CodeRetrieverEvaluation Protocol=Zero-Shot2023.05 | — | 68.7 | 63.7 | 87.6 | 67.7 | 69 | 62.8 | 69.1 | |
| CodeRetriever (Hard Negative)Evaluation Protocol=Fine-Tuning2023.05 | — | 75.1 | 69.8 | 92.3 | 74 | 74.9 | 69.1 | 75.9 | |
| CodeRetriever (Inbatch)Evaluation Protocol=Fine-Tuning2023.05 | — | 75.3 | 69.5 | 91.6 | 73.3 | 74 | 68.2 | 75.3 | |
| CodeT5Evaluation Protocol=Fine-Tuning2023.05 | — | 71.9 | 65.5 | 88.8 | 69.8 | 68.6 | 64.5 | 71.5 | |
| GraphCodeBERTEvaluation Protocol=Zero-Shot2023.05 | — | 1.5 | 0.4 | 0.2 | 0.4 | 0.7 | 2.1 | 0.88 | |
| GraphCodeBERTEvaluation Protocol=Fine-Tuning2023.05 | — | 70.3 | 64.4 | 89.7 | 69.2 | 69.1 | 64.9 | 71.3 | |
| SANTAEvaluation Protocol=Zero-Shot2023.05 | — | 72.6 | 62.4 | 88.9 | 70 | 68.6 | 62.8 | 70.9 | |
| SANTA (Hard Negative)Evaluation Protocol=Fine-Tuning2023.05 | — | 74.7 | 68.6 | 91.8 | 73.7 | 73.7 | 68.6 | 75.2 |