Tool Retrieval on ToolLens (test)
96.3Recall@3L2-TR
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| L2-TREmbedding=Trained, Regularization=L22026.06 | 96.3 | 98.4 | — | — | — | — | |
| L1-TREmbedding=Trained, Regularization=L12026.06 | 96.2 | 98.2 | — | — | — | — | |
| NNN-TREmbedding=Trained2026.06 | 96.1 | 98.4 | — | — | — | — | |
| COLTBackbone=Contriever, Framework=+COLT (Ours)2024.05 | 93.64 | 97.75 | 94.53 | 96.91 | 84.55 | 94.08 | |
| COLTBackbone=coCondensor, Framework=+COLT (Ours)2024.05 | 92.65 | 97.78 | 93.16 | 96.17 | 82.25 | 94.56 | |
| COLTBackbone=ANCE, Framework=+COLT (Ours)2024.05 | 92.15 | 97.78 | 92.78 | 96.1 | 80.5 | 94.4 | |
| COLTBackbone=TAS-B, Framework=+COLT (Ours)2024.05 | 91.49 | 96.91 | 92.48 | 95.63 | 79 | 92.22 | |
| COLT2026.06 | 89.1 | 97 | — | — | — | — | |
| MMR2026.06 | 88.8 | 97.1 | — | — | — | — | |
| L1-FIXEmbedding=Frozen, Regularization=L12026.06 | 88.5 | 95.6 | — | — | — | — | |
| NNN-FIXEmbedding=Frozen2026.06 | 88 | 96.1 | — | — | — | — | |
| L2-FIXEmbedding=Frozen, Regularization=L22026.06 | 87.5 | 96 | — | — | — | — | |
| ContrieverFramework=+Fine-tune2024.05 | 83.58 | 95.17 | 84.98 | 91.69 | 59.46 | 88.65 | |
| coCondensorFramework=+Fine-tune2024.05 | 82.37 | 94.69 | 83.9 | 91.06 | 56.37 | 86.73 | |
| TAS-BFramework=+Fine-tune2024.05 | 81.26 | 94.06 | 82.54 | 89.94 | 54.66 | 85.72 | |
| DENSE2026.06 | 81.2 | 92.5 | — | — | — | — | |
| ANCEFramework=+Fine-tune2024.05 | 80.62 | 94.17 | 82.35 | 90.15 | 54.23 | 85.83 | |
| ContrieverFramework=Zero-shot2024.05 | 25.67 | 31.15 | 26.96 | 29.95 | 7.46 | 9.8 | |
| BM25Backbone=BM252024.05 | 21.58 | 26.88 | 23.19 | 26.09 | 3.89 | 6.13 | |
| ANCEFramework=Zero-shot2024.05 | 20.82 | 26.56 | 21.45 | 24.57 | 5.06 | 7.46 | |
| TAS-BFramework=Zero-shot2024.05 | 19.1 | 23.71 | 19.81 | 22.33 | 5.17 | 7.14 | |
| coCondensorFramework=Zero-shot2024.05 | 15.33 | 19.37 | 16.15 | 18.32 | 3.02 | 5.33 | |
| COLT2026.06 | — | — | — | — | 73.5 | 93.6 | |
| DENSE2026.06 | — | — | — | — | 55.2 | 81.5 | |
| L1-FIXEmbedding status=Fixed, Regularization=L12026.06 | — | — | — | — | 73.4 | 90 | |
| L1-TREmbedding status=Trained, Regularization=L12026.06 | — | — | — | — | 92.1 | 96.3 | |
| L2-FIXEmbedding status=Fixed, Regularization=L22026.06 | — | — | — | — | 70.5 | 91.7 | |
| L2-TREmbedding status=Trained, Regularization=L22026.06 | — | — | — | — | 92.4 | 97 | |
| MMR2026.06 | — | — | — | — | 72.2 | 93.1 | |
| NNN-FIXEmbedding status=Fixed, Regularization=Elastic Net2026.06 | — | — | — | — | 72 | 91.4 | |
| NNN-TREmbedding status=Trained, Regularization=Elastic Net2026.06 | — | — | — | — | 91.8 | 97 |