Tool Selection on APIGen sampled (test)
99.4Tool Selection F1 (2 calls)TokMem (w/ adapt)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| TokMem (w/ adapt)Model=Llama 3.1 8B, #Params=0.20M2025.10 | 99.4 | 97.9 | 100 | 99.1 | |
| TokMem (w/ adapt)Model=Llama 3.2 3B, #Params=0.15M2025.10 | 99.2 | 98.2 | 100 | 99.2 | |
| Fine-TuingModel=Llama 3.1 8B, #Params=3.41M2025.10 | 98.8 | 97.2 | 98.2 | 98.1 | |
| Fine-TuingModel=Llama 3.2 3B, #Params=2.29M2025.10 | 98.7 | 98.1 | 96.8 | 97.9 | |
| TokMem (w/ adapt)Model=Llama 3.2 1B, #Params=0.10M2025.10 | 98.4 | 98 | 98.9 | 98.4 | |
| TokMem (w/o adapt)Model=Llama 3.2 1B, #Params=0.10M2025.10 | 86.8 | 80.9 | 90.8 | 86.2 | |
| TokMem (w/o adapt)Model=Llama 3.1 8B, #Params=0.20M2025.10 | 84.9 | 82 | 81.6 | 82.8 | |
| TokMem (w/o adapt)Model=Llama 3.2 3B, #Params=0.15M2025.10 | 82.6 | 79.3 | 67.2 | 76.4 | |
| ICLModel=Llama 3.1 8B, #Params=–2025.10 | 79.7 | 72.9 | 75.4 | 76 | |
| RAGModel=Llama 3.1 8B, #Params=–2025.10 | 79.6 | 75.3 | 93 | 82.6 | |
| RAGModel=Llama 3.2 3B, #Params=–2025.10 | 78.1 | 71.2 | 69.3 | 72.8 | |
| ICLModel=Llama 3.2 3B, #Params=–2025.10 | 66.8 | 59.2 | 59.6 | 61.9 | |
| RAGModel=Llama 3.2 1B, #Params=–2025.10 | 29.5 | 10.8 | 10.5 | 16.9 | |
| ICLModel=Llama 3.2 1B, #Params=–2025.10 | 27.6 | 11.1 | 10.5 | 16.4 | |
| Fine-TuingModel=Llama 3.2 1B, #Params=0.85M2025.10 | 10.4 | 9.5 | 7 | 9 |