Tool retrieval on ToolRet
45.96Avg nDCG@10gte-qwen2-1.5B
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gte-qwen2-1.5BTop-K=102026.02 | 45.96 | 37.53 | 9.31 | 48.31 | 30.95 | 47.38 | 7.29 | 61.12 | 59.55 | 52.98 | 10.63 | 59.47 | 45.68 | 45.39 | — | — | — | |
| pplx-embed-v1-4BQuantization (e.g. INT8)=INT8, Top-K (k=10)=102026.02 | 44.45 | 42.07 | 9.89 | 52.55 | 36.42 | 41.61 | 6.2 | 54.96 | 53.12 | 49.68 | 9.53 | 57.77 | 45.98 | 45.17 | — | — | — | |
| pplx-embed-v1-0.6BQuantization (e.g. INT8)=INT8, Top-K (k=10)=102026.02 | 43.05 | 35.84 | 8.19 | 45.26 | 30.31 | 45.52 | 6.62 | 59.07 | 57.32 | 47.79 | 9.24 | 55.33 | 45.24 | 44.29 | — | — | — | |
| nv-embed-v1Top-K=102026.02 | 42.71 | 31.51 | 7.74 | 40.52 | 26.74 | 47.92 | 7.1 | 62.07 | 59.6 | 48.7 | 10.07 | 57.69 | 43.88 | 43.41 | — | — | — | |
| gritlm-7BTop-K=102026.02 | 41.13 | 36.58 | 9.34 | 46.01 | 27.65 | 41.26 | 6.17 | 53.81 | 52.07 | 45.55 | 9.74 | 54.01 | 41.4 | 40.37 | — | — | — | |
| Qwen3 (QTND)Train?=Yes, Input=QTND2025.10 | 39.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.28 | 51.99 | 39.5 | |
| e5-mistral-7BTop-K=102026.02 | 38.97 | 31.07 | 7.65 | 41.3 | 27.04 | 44.97 | 6.66 | 58.95 | 56.79 | 40.88 | 7.91 | 49.35 | 38.35 | 40.73 | — | — | — | |
| bge-largeTop-K=102026.02 | 38.49 | 30.03 | 7.01 | 39.28 | 25.63 | 41.53 | 6 | 52.76 | 51.18 | 43.9 | 8.31 | 51.79 | 42.24 | 39.68 | — | — | — | |
| Qwen3 (Q) (TR)Train?=Yes, Protocol=ToolRet (TR) baseline, Input=Question (Q)2025.10 | 38.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.8 | 49.91 | 39.07 | |
| Qwen3 (TND)Train?=Yes, Input=TND2025.10 | 37.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.2 | 51.23 | 36.98 | |
| Qwen3 (QTND)Train?=No, Input=QTND2025.10 | 37 | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.55 | 48.93 | 37.68 | |
| gte-baseTop-K=102026.02 | 36.79 | 30.75 | 7 | 39.44 | 25.88 | 41.68 | 6.2 | 53.96 | 51.64 | 37.95 | 6.96 | 46.57 | 38.1 | 38.54 | — | — | — | |
| bm25Top-K=102026.02 | 36.46 | 26.33 | 6.1 | 34.22 | 22.79 | 41.9 | 6.2 | 56.49 | 55.39 | 41.16 | 8.39 | 48.6 | 38.9 | 39.03 | — | — | — | |
| Qwen3 (TND) (K)Train?=No, Input=TND, Source=K2025.10 | 35.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.93 | 50.15 | 35.5 | |
| Qwen3 (0s)Train?=No, Protocol=zero-shot2025.10 | 35.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.2 | 47.09 | 36.64 | |
| gtr-t5-largeTop-K=102026.02 | 34.39 | 24.37 | 5.27 | 31.64 | 21.26 | 36.76 | 5.33 | 47.42 | 45.92 | 42.04 | 8.48 | 50.84 | 40 | 35.73 | — | — | — | |
| COLT (Phase-1)Train?=Yes, Protocol=Phase-12025.10 | 33.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.06 | 38.8 | 36.15 | |
| BM25 (QTND)Train?=No, Input=QTND2025.10 | 31.41 | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.82 | 42.83 | 31.72 | |
| BM25 (TND) (K)Train?=No, Input=TND, Source=K2025.10 | 28.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.31 | 39.85 | 28.29 | |
| BM25 (0s)Train?=No, Protocol=zero-shot2025.10 | 27.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | 5.72 | 36.25 | 28.1 | |
| COLT (Phase-2)Train?=Yes, Protocol=Phase-22025.10 | 0.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.21 | — |