Android Device Automation on AndroidLab (test)
0.3623SRDeepSeek-V3.1
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DeepSeek-V3.1Type=Prompting (PT), Input Mode=XML2025.12 | 0.3623 | 0.4095 | 81.01 | 0.9463 | |
| Qwen3-8B-InstructOperating Mode=ReAct, Type=Reinforcement Learning (RL), Source=ours, Input Mode=XML2025.12 | 0.3623 | 0.4196 | 88.04 | 0.9449 | |
| Qwen3-235B-A22BType=Prompting (PT), Input Mode=XML2025.12 | 0.3478 | 0.3876 | 83.35 | 0.8948 | |
| Qwen3-32B-InstructOperating Mode=ReAct, Type=Reinforcement Learning (RL), Source=ours, Input Mode=XML2025.12 | 0.3478 | 0.4026 | 89.47 | 0.9367 | |
| GPT-4-1106-PreviewType=Prompting (PT), Input Mode=XML2025.12 | 0.3116 | 0.3821 | 66.34 | 0.8624 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Reinforcement Learning (RL), Source=ours, Input Mode=XML2025.12 | 0.3115 | 0.3803 | 81.28 | 0.958 | |
| Qwen2.5-7B-InstructOperating Mode=ReAct, Type=Reinforcement Learning (RL), Source=ours, Input Mode=XML2025.12 | 0.3043 | 0.352 | 102.3 | 0.9636 | |
| Qwen2.5-7B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Source=ours, Input Mode=XML2025.12 | 0.3015 | 0.3659 | 49.19 | 0.7328 | |
| Qwen3-32B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Source=ours, Input Mode=XML2025.12 | 0.2898 | 0.3592 | 97.79 | 0.9733 | |
| GLM4-PlusType=Prompting (PT), Input Mode=XML2025.12 | 0.2754 | 0.3208 | 92.35 | 0.8341 | |
| Qwen3-8B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Source=ours, Input Mode=XML2025.12 | 0.2681 | 0.3109 | 72.16 | 0.6985 | |
| GPT-4oType=Prompting (PT), Input Mode=XML2025.12 | 0.2536 | 0.3056 | 107.45 | 0.8656 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Fine-tuning (FT), Note=Official results trained on Android Instruct, Input Mode=XML2025.12 | 0.2391 | 0.3031 | 75.58 | 0.9246 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Fine-tuning (FT), Source=ours, Input Mode=XML2025.12 | 0.2391 | 0.3036 | 37.96 | 0.8323 | |
| Qwen3-32B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Dataset=Android Instruct [1], Input Mode=XML2025.12 | 0.2246 | 0.282 | 39.28 | 0.655 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Fine-tuning (FT), Dataset=Android Instruct [1], Input Mode=XML2025.12 | 0.2028 | 0.2613 | 69.44 | 0.9043 | |
| Qwen2.5-7B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Dataset=Android Instruct [1], Input Mode=XML2025.12 | 0.2028 | 0.2705 | 35.52 | 0.6246 | |
| Qwen3-8B-InstructOperating Mode=ReAct, Type=Fine-tuning (FT), Dataset=Android Instruct [1], Input Mode=XML2025.12 | 0.1956 | 0.256 | 38.69 | 0.6518 | |
| Gemini-1.5-ProType=Prompting (PT), Input Mode=XML2025.12 | 0.1884 | 0.224 | 57.72 | 0.8399 | |
| Qwen3-32B-InstructOperating Mode=ReAct, Type=Prompting (PT), Input Mode=XML2025.12 | 0.1812 | 0.218 | 91.99 | 0.8757 | |
| Qwen2.5-7B-InstructOperating Mode=ReAct, Type=Prompting (PT), Input Mode=XML2025.12 | 0.1232 | 0.1498 | 67.56 | 0.7852 | |
| Qwen3-8B-InstructOperating Mode=ReAct, Type=Prompting (PT), Input Mode=XML2025.12 | 0.1014 | 0.1238 | 66.21 | 0.6715 | |
| Gemini-1.00Type=Prompting (PT), Input Mode=XML2025.12 | 0.087 | 0.1075 | 51.8 | 0.7108 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Prompting (PT), Input Mode=XML2025.12 | 0.0507 | 0.0628 | 52.77 | 0.5182 | |
| LLaMA3.1-8B-InstructOperating Mode=Act-only, Type=Prompting (PT), Note=Official results, Input Mode=XML2025.12 | 0.0217 | 0.0362 | — | 0.5277 |