Tool selection on MetaTool similar choices subtask (test)
83.4AccuracyOATS-S1
Evaluation results
| Method | Links | |
|---|---|---|
| OATS-S1Latency=3.5 ms, Hardware=CPU, type=emb. refine2026.03 | 83.4 | |
| Vicuna-7bLatency=∼2–5 s, Hardware=GPU2026.03 | 73.5 | |
| ChatGPT (GPT-3.5)Latency=∼1–3 s, Hardware=API / GPU2026.03 | 69.1 | |
| Static Embedding (SE)Latency=3.7 ms, Hardware=CPU2026.03 | 66.4 | |
| Vicuna-13bLatency=∼3–8 s, Hardware=GPU2026.03 | 58.2 | |
| Average (9 LLMs)Hardware=GPU2026.03 | 57 | |
| LLaMA2-13bLatency=∼2–5 s, Hardware=GPU2026.03 | 44.1 | |
| BM25Latency=0.4 ms, Hardware=CPU2026.03 | 42.7 |