Step success rate on AndroidControl (IDD)
69.4Step Success Rate (SSR)T3A* (Qwen2.5-7B-Instruct-ASL)
Evaluation results
| Method | Links | |
|---|---|---|
| T3A* (Qwen2.5-7B-Instruct-ASL)Type=Fine-Tuned, Agent=T3A*, Model=Qwen2.5-7B-Instruct-ASL (Ours), Input Modality=Text2025.06 | 69.4 | |
| T3A* (Qwen2.5-7B-Instruct-SFT)Type=Fine-Tuned, Agent=T3A*, Model=Qwen2.5-7B-Instruct-SFT, Input Modality=Text2025.06 | 67.4 | |
| M3A (GPT-4o)Type=Prompt-Driven, Agent=M3A, Model=GPT-4o, Input Modality=Image + Text2025.06 | 60.8 | |
| T3A (GPT-4o)Type=Prompt-Driven, Agent=T3A, Model=GPT-4o, Input Modality=Text2025.06 | 56.1 | |
| M3A (Gemini-2.5-Flash)Type=Prompt-Driven, Agent=M3A, Model=Gemini-2.5-Flash, Input Modality=Image + Text2025.06 | 49.4 | |
| T3A (Gemini-2.5-Flash)Type=Prompt-Driven, Agent=T3A, Model=Gemini-2.5-Flash, Input Modality=Text2025.06 | 49.1 | |
| T3A* (Gemini-2.5-Flash)Type=Prompt-Driven, Agent=T3A*, Model=Gemini-2.5-Flash, Input Modality=Text2025.06 | 46.8 | |
| SeeAct (Gemini-2.5-Flash)Type=Prompt-Driven, Agent=SeeAct, Model=Gemini-2.5-Flash, Input Modality=Image + Text2025.06 | 36.7 | |
| SeeAct (GPT-4o)Type=Prompt-Driven, Agent=SeeAct, Model=GPT-4o, Input Modality=Image + Text2025.06 | 31.5 | |
| T3A* (Qwen2.5-7B-Instruct)Type=Prompt-Driven, Agent=T3A*, Model=Qwen2.5-7B-Instruct, Input Modality=Text2025.06 | 26.1 |