Agentic Phone Use on 150-task Real-phone Human Evaluation Single-App
62Task Success RatePhoneBuddy-4B-Real+Mock
Evaluation results
| Method | Links | |
|---|---|---|
| PhoneBuddy-4B-Real+MockParameters=4B, Training=mixed RL in real-app and mock-app2026.06 | 62 | |
| Seed 2.0 Pro2026.06 | 60 | |
| Gemini 3.1 Pro2026.06 | 58 | |
| PhoneBuddy-4B-Real+MockParameters=4B, Training=mixed RL in real-app and mock-app2026.06 | 56 | |
| PhoneBuddy-4B-RealParameters=4B, Training=real-app RL2026.06 | 54 | |
| PhoneBuddy-4B-SFTParameters=4B, Training=supervised fine-tuning baseline2026.06 | 54 | |
| Gemini 3.1 Pro2026.06 | 50 | |
| GPT-5.42026.06 | 50 | |
| PhoneBuddy-4B-RealParameters=4B, Training=real-app RL2026.06 | 48 | |
| Seed 2.0 Pro2026.06 | 44 | |
| GPT-5.42026.06 | 40 | |
| Claude Opus 4.72026.06 | 38 | |
| PhoneBuddy-4B-SFTParameters=4B, Training=supervised fine-tuning baseline2026.06 | 34 | |
| Claude Opus 4.72026.06 | 28 |