Function Calling on Mobile Actions
84.5Overall AccuracyST-Qwen3-4B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ST-Qwen3-4BSpeedup ratio=3.9×2026.02 | 84.5 | — | — | — | 99.9 | |
| ST-Qwen2.5-7BSpeedup ratio=4.5×2026.02 | 83.2 | — | — | — | 100 | |
| ST-Qwen2.5-3BSpeedup ratio=3.8×2026.02 | 81.4 | — | — | — | 99.9 | |
| ST-Qwen2.5-14BSpeedup ratio=4.4×2026.02 | 80.5 | — | — | — | 100 | |
| ST-Qwen2.5-1.5BSpeedup ratio=3.5×2026.02 | 72.8 | — | — | — | 100 | |
| ST-Qwen2.5-0.5BSpeedup ratio=3.1×2026.02 | 69.3 | — | — | — | 99.3 | |
| Qwen2.5-7BSpeedup ratio=1×2026.02 | 68.4 | — | — | — | 77.2 | |
| Qwen3-4BSpeedup ratio=1×2026.02 | 68 | — | — | — | 75 | |
| Qwen2.5-14BSpeedup ratio=1×2026.02 | 67 | — | — | — | 75 | |
| Qwen2.5-3BSpeedup ratio=1×2026.02 | 66.2 | — | — | — | 75 | |
| Qwen2.5-1.5BSpeedup ratio=1×2026.02 | 65.5 | — | — | — | 74.9 | |
| Qwen2.5-0.5BSpeedup ratio=1×2026.02 | 56.7 | — | — | — | 74.4 | |
| FunctionGemma (270M)Framework=vLLM, Hardware=RTX 40902026.02 | — | 61.1 | 139.5 | 7.7 | — | |
| Qwen2.5-0.5B (Baseline)Framework=vLLM, Hardware=RTX 40902026.02 | — | 110.1 | 154 | 4.3 | — | |
| Qwen3-4B (Baseline)Framework=vLLM, Hardware=RTX 40902026.02 | — | 468.5 | 667 | 1 | — | |
| ST-Qwen-0.5BFramework=vLLM, Hardware=RTX 40902026.02 | — | 51 | 74.5 | 9.2 | — |