Natural Language Code Search on CodeSearchNet
82.3Overall ScoreEttin-Enc-1B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Ettin-Enc-1BModel Size Category=XL, Parameters=884M-1.2B2025.07 | 82.3 | — | — | — | — | — | — | |
| Ettin-Enc-400mModel Size Category=Large, Parameters=353-395M2025.07 | 80.7 | — | — | — | — | — | — | |
| ModernBERT largeModel Size Category=Large, Parameters=353-395M2025.07 | 78.3 | — | — | — | — | — | — | |
| GraphCodeBERTCandidate Codes=10002020.09 | 77.4 | 73.2 | 71.1 | 84.1 | 87.9 | 75.7 | 72.5 | |
| Ettin-Enc-150mModel Size Category=Base, Parameters=123-150M2025.07 | 76.3 | — | — | — | — | — | — | |
| CodeBERTCandidate Codes=10002020.09 | 76 | 69.3 | 70.6 | 84 | 86.9 | 74.8 | 70.6 | |
| ModernBERT baseModel Size Category=Base, Parameters=123-150M2025.07 | 75.9 | — | — | — | — | — | — | |
| DeBERTa-v1-xlModel Size Category=XL, Parameters=884M-1.2B2025.07 | 75.6 | — | — | — | — | — | — | |
| Ettin-Enc-68mModel Size Category=Small, Parameters=68-82M2025.07 | 75.1 | — | — | — | — | — | — | |
| RoBERTa (code)Candidate Codes=10002020.09 | 72.6 | 66.1 | 64 | 81.9 | 84.4 | 72.1 | 67.1 | |
| EuroBERTNumber of Parameters=2.1B2025.03 | 72.6 | — | — | — | — | — | — | |
| EuroBERTNumber of Parameters=610M2025.03 | 69.9 | — | — | — | — | — | — | |
| RoBERTaCandidate Codes=10002020.09 | 69.7 | 62.5 | 60.6 | 82 | 80.9 | 66.6 | 65.8 | |
| Ettin-Enc-32mModel Size Category=XS, Parameters=28-33M2025.07 | 69.2 | — | — | — | — | — | — | |
| ModernBERTNumber of Parameters=395M2025.03 | 65.8 | — | — | — | — | — | — | |
| DistilRoBERTaModel Size Category=Small, Parameters=68-82M2025.07 | 60.3 | — | — | — | — | — | — | |
| Ettin-Enc-17mModel Size Category=XXS, Parameters=7-17M2025.07 | 59.1 | — | — | — | — | — | — | |
| EuroBERTNumber of Parameters=210M2025.03 | 58.9 | — | — | — | — | — | — | |
| selfAttCandidate Codes=10002020.09 | 56.3 | 36.5 | 45.1 | 68.1 | 69.2 | 58.7 | 60.1 | |
| BERT largeModel Size Category=Large, Parameters=353-395M2025.07 | 54.4 | — | — | — | — | — | — | |
| XLM-RoBERTaNumber of Parameters=3.5B2025.03 | 54.1 | — | — | — | — | — | — | |
| ModernBERTNumber of Parameters=150M2025.03 | 53.9 | — | — | — | — | — | — | |
| NBowCandidate Codes=10002020.09 | 51.8 | 42.9 | 46.1 | 64.1 | 58.1 | 51.4 | 48.4 | |
| BERT baseModel Size Category=Base, Parameters=123-150M2025.07 | 51 | — | — | — | — | — | — | |
| MiniLM L12Model Size Category=XS, Parameters=28-33M2025.07 | 48.3 | — | — | — | — | — | — | |
| DistilBERTModel Size Category=Small, Parameters=68-82M2025.07 | 47.9 | — | — | — | — | — | — | |
| CNNCandidate Codes=10002020.09 | 47.5 | 24.5 | 35.2 | 62.7 | 57.1 | 52.7 | 52.9 | |
| BERT-smallModel Size Category=XS, Parameters=28-33M2025.07 | 46 | — | — | — | — | — | — | |
| BERT-miniModel Size Category=XXS, Parameters=7-17M2025.07 | 41.3 | — | — | — | — | — | — | |
| XLM-RoBERTaNumber of Parameters=560M2025.03 | 40.8 | — | — | — | — | — | — | |
| TinyBERTModel Size Category=XXS, Parameters=7-17M2025.07 | 39.8 | — | — | — | — | — | — | |
| mGTENumber of Parameters=305M2025.03 | 34 | — | — | — | — | — | — | |
| BiRNNCandidate Codes=10002020.09 | 25.8 | 8.4 | 15.3 | 45.2 | 32.1 | 28.7 | 25.1 | |
| XLM-RoBERTaNumber of Parameters=280M2025.03 | 23 | — | — | — | — | — | — | |
| mDeBERTaNumber of Parameters=280M2025.03 | 2.8 | — | — | — | — | — | — |