Adversarial Attack against SeeAct agent on Mind2Web 600 tasks (test)
54.1ASR Finance (pass@10)Genesis
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GenesisBackend LLM=GPT-4o2025.10 | 54.1 | 55.3 | 43.9 | 58.7 | 53 | |
| GenesisBackend LLM=GPT-4o, Initialization Setting=w/o Initialization2025.10 | 44.1 | 43.8 | 43.7 | 46.3 | 44.5 | |
| GenesisBackend LLM=Gemini-2.5-Flash2025.10 | 43.9 | 44 | 45.2 | 46.6 | 44.9 | |
| AdvAgentBackend LLM=GPT-4o2025.10 | 42.8 | 43.5 | 44.1 | 44 | 43.6 | |
| GenesisBackend LLM=Gemini-2.5-Flash, Initialization Setting=w/o Initialization2025.10 | 34.9 | 35.7 | 36.2 | 37.5 | 36.1 | |
| AdvAgentBackend LLM=Gemini-2.5-Flash2025.10 | 34.8 | 34.3 | 35.4 | 35.5 | 35 | |
| EIABackend LLM=GPT-4o2025.10 | 34.2 | 34.3 | 35.5 | 35.3 | 34.8 | |
| EIABackend LLM=Gemini-2.5-Flash2025.10 | 27.5 | 27.6 | 28.4 | 28.2 | 27.9 | |
| InjecAttackBackend LLM=GPT-4o2025.10 | 24 | 25.3 | 25.4 | 26.5 | 25.3 | |
| GenesisBackend LLM=GPT-52025.10 | 23 | 23.5 | 23.8 | 25.7 | 24 | |
| InjecAttackBackend LLM=Gemini-2.5-Flash2025.10 | 19.2 | 20.4 | 20.5 | 21.1 | 20.3 | |
| GenesisBackend LLM=GPT-5, Initialization Setting=w/o Initialization2025.10 | 19.1 | 18.7 | 19.7 | 19.8 | 19.3 | |
| AdvAgentBackend LLM=GPT-52025.10 | 18.1 | 24.1 | 18.6 | 19.3 | 20 | |
| EIABackend LLM=GPT-52025.10 | 14.8 | 14.6 | 15.2 | 15.1 | 14.9 | |
| AgentAttackBackend LLM=GPT-4o2025.10 | 14.7 | 14.1 | 15.9 | 15.8 | 15.1 | |
| AgentAttackBackend LLM=Gemini-2.5-Flash2025.10 | 11.8 | 11.4 | 12.8 | 12.7 | 12.2 | |
| InjecAttackBackend LLM=GPT-52025.10 | 10.1 | 10.9 | 10.8 | 11.4 | 10.8 | |
| AgentAttackBackend LLM=GPT-52025.10 | 6.4 | 6 | 6.9 | 6.8 | 6.5 | |
| I-GCGBackend LLM=GPT-4o2025.10 | 3.5 | 3.7 | 4.2 | 4.6 | 4 | |
| I-GCGBackend LLM=Gemini-2.5-Flash2025.10 | 2.4 | 3.3 | 3 | 4.1 | 3.2 | |
| GCGBackend LLM=Gemini-2.5-Flash2025.10 | 1.3 | 1.5 | 2.1 | 2.4 | 1.8 | |
| I-GCGBackend LLM=GPT-52025.10 | 1.2 | 2 | 1.7 | 2.6 | 1.9 | |
| GCGBackend LLM=GPT-4o2025.10 | 1.1 | 2.5 | 2.2 | 3.4 | 2.3 | |
| GCGBackend LLM=GPT-52025.10 | 0.7 | 0.9 | 1.5 | 1.8 | 1.2 |