Financial Tool-Use on FinToolBench
62.15Average Success RateGemini-2.5-Pro
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini-2.5-ProEvaluation Mode=Prompt Mode, Model Category=Closed-Source LLMs2026.03 | 62.15 | 81.56 | 72.34 | 67.32 | 67.27 | 54.64 | 25 | 66.9 | 14.81 | 72.75 | |
| Gemini-2.5-ProEvaluation Mode=Function Calling (FC) Mode, Model Category=Closed-Source LLMs2026.03 | 60.98 | 64.77 | 47.68 | 49.56 | 51.17 | 85.07 | 40.75 | 87.83 | 34.78 | 53.94 | |
| GPT-4oEvaluation Mode=Prompt Mode, Model Category=Closed-Source LLMs2026.03 | 59.67 | 74.87 | 58.02 | 62.36 | 53.98 | 65.47 | 28.85 | 74.18 | 37.04 | 61.99 | |
| Gemini-2.5-FlashEvaluation Mode=Function Calling (FC) Mode, Model Category=Closed-Source LLMs2026.03 | 58.35 | 57.73 | 28.5 | 42.8 | 38.8 | 94.27 | 50.37 | 96 | 43.48 | 41.66 | |
| GPT-4oEvaluation Mode=Function Calling (FC) Mode, Model Category=Closed-Source LLMs2026.03 | 58.26 | 74.53 | 53.25 | 71.56 | 56.1 | 63.01 | 6.25 | 83.13 | 11.11 | 62.59 | |
| Qwen3-235B-InstructEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 58.06 | 76.98 | 59.83 | 64.04 | 67.14 | 59.47 | 9.62 | 69.37 | 3.7 | 66.61 | |
| Claude-4.0-SonnetEvaluation Mode=Prompt Mode, Model Category=Closed-Source LLMs2026.03 | 57.96 | 58.09 | 37.59 | 37.09 | 43.98 | 100 | 31.73 | 97.25 | 37.04 | 44.03 | |
| FinTool-Qwen3-14BEvaluation Mode=Prompt Mode, Model Category=FinTool-Tuned (Ours)2026.03 | 57.95 | 77.23 | 58.24 | 79.99 | 68.78 | 30.32 | 50 | 41.07 | 74.07 | 68.82 | |
| FinTool-Qwen3-14BEvaluation Mode=Function Calling (FC) Mode, Model Category=FinTool-Tuned (Ours)2026.03 | 57.77 | 77.86 | 59.11 | 76.98 | 66.18 | 33.32 | 45.19 | 45.74 | 74.07 | 68.17 | |
| FinTool-Qwen3-8BEvaluation Mode=Function Calling (FC) Mode, Model Category=FinTool-Tuned (Ours)2026.03 | 57.64 | 77.2 | 56.01 | 74.53 | 65.5 | 42.98 | 44.23 | 42.99 | 59.26 | 66.26 | |
| Qwen2.5-14B-InstructEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 57.23 | 70.12 | 54.23 | 56.45 | 53.12 | 78.34 | 7.12 | 81.23 | 4.1 | 57.5 | |
| Claude-4.0-SonnetEvaluation Mode=Function Calling (FC) Mode, Model Category=Closed-Source LLMs2026.03 | 55.93 | 67.28 | 62.15 | 68.27 | 63.36 | 44.7 | 32.65 | 53.08 | 44.44 | 65.33 | |
| Qwen2.5-14B-InstructEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 55.75 | 68.47 | 52.06 | 54.91 | 51.93 | 76.74 | 6.73 | 79.4 | 3.7 | 56.99 | |
| FinTool-Qwen3-8BEvaluation Mode=Prompt Mode, Model Category=FinTool-Tuned (Ours)2026.03 | 55.68 | 78.1 | 58.12 | 79.37 | 67.82 | 30.32 | 42.31 | 33.72 | 70.37 | 67.79 | |
| Deepseek-V3.1-TerminusEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 55.32 | 79.83 | 56.83 | 73.61 | 62.82 | 47.81 | 3.85 | 62.5 | 3.7 | 67.04 | |
| Qwen2.5-72B-InstructEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 53.23 | 79.12 | 61.23 | 66.45 | 64.12 | 38.23 | 6.12 | 57.34 | 11.5 | 67.1 | |
| Qwen3-235B-InstructEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 53.05 | 77.19 | 59.11 | 67.79 | 66.5 | 43.58 | 1.92 | 55.22 | 0 | 67.02 | |
| Qwen2.5-32B-InstructEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 52.85 | 68.23 | 47.12 | 55.45 | 53.12 | 67.34 | 4.12 | 74.56 | 4.1 | 54.2 | |
| Qwen2.5-72B-InstructEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 51.86 | 78.12 | 59.69 | 64.79 | 62.84 | 36.15 | 5.77 | 55.63 | 11.11 | 66.92 | |
| Qwen3-32BEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 51.67 | 79.82 | 57.85 | 63.6 | 64.79 | 36.92 | 3.85 | 54.88 | 0 | 66.2 | |
| Qwen2.5-32B-InstructEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 51.18 | 66.26 | 45.39 | 53.43 | 51.33 | 65.08 | 3.85 | 72.94 | 3.7 | 53.81 | |
| Qwen3-14BEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 50.47 | 68.25 | 51.13 | 43.64 | 44.39 | 63.75 | 4.81 | 77.34 | 0 | 53.13 | |
| Qwen3-8BEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 50.05 | 66.55 | 43.37 | 39.22 | 42.97 | 73.42 | 3.85 | 80.98 | 3.7 | 49.05 | |
| Qwen3-32BEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 49.58 | 73.13 | 50.14 | 48.23 | 51.49 | 53.09 | 3.85 | 67.1 | 0 | 54.71 | |
| Qwen3-14BEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 47.87 | 67.43 | 51.69 | 64.68 | 58.58 | 36.21 | 1.92 | 54.6 | 0 | 58.59 | |
| Qwen3-8BEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 47.55 | 70.24 | 47.04 | 59.42 | 53.71 | 46.09 | 1.92 | 54.46 | 3.7 | 57.37 | |
| GPT-3.5-TurboEvaluation Mode=Function Calling (FC) Mode, Model Category=Closed-Source LLMs2026.03 | 43.67 | 68.51 | 37.15 | 53.36 | 45.78 | 44.78 | 4.17 | 51.91 | 0 | 48.94 | |
| Qwen2.5-7B-InstructEvaluation Mode=Function Calling (FC) Mode, Model Category=Open-Source LLMs2026.03 | 43.23 | 63.12 | 46.23 | 62.45 | 49.12 | 32.34 | 1.12 | 48.23 | 1.1 | 52.1 | |
| Qwen2.5-7B-InstructEvaluation Mode=Prompt Mode, Model Category=Open-Source LLMs2026.03 | 41.67 | 61.1 | 45.12 | 60.17 | 47.93 | 30.81 | 0 | 46.57 | 0 | 51.67 | |
| Gemini-2.5-FlashEvaluation Mode=Prompt Mode, Model Category=Closed-Source LLMs2026.03 | 40.85 | 1.92 | 4.6 | 22.22 | 18.71 | 100 | 38.46 | 100 | 25.93 | 10.25 | |
| GPT-3.5-TurboEvaluation Mode=Prompt Mode, Model Category=Closed-Source LLMs2026.03 | 38.47 | 56.77 | 39.48 | 42.75 | 37.26 | 41.72 | 4.81 | 46.5 | 11.11 | 45.04 |