Indirect Prompt Injection Red-teaming on RocketChat
3.33ASR (Confidentiality)Operator
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OperatorAgent Category=Specialized Computer-Use Agents2025.05 | 3.33 | 6.55 | 6.25 | 8.33 | 13.1 | 18.75 | |
| Claude 3.5 SonnetAgent Category=Adapted LLM-based Agents2025.05 | 8.33 | 73.21 | 43.75 | 96.67 | 82.14 | 75 | |
| Claude 3.5 Sonnet | CUAAgent Category=Specialized Computer-Use Agents2025.05 | 11.67 | 50 | 6.25 | 96.67 | 86.31 | 70.83 | |
| Operator (w/o checks)Agent Category=Specialized Computer-Use Agents2025.05 | 21.67 | 48.81 | 37.5 | 73.33 | 59.52 | 64.58 | |
| Claude 3.7 Sonnet | CUAAgent Category=Specialized Computer-Use Agents2025.05 | 26.67 | 60.12 | 43.75 | 81.67 | 72.62 | 68.75 | |
| GPT-4oAgent Category=Adapted LLM-based Agents2025.05 | 30 | 95.24 | 58.33 | 100 | 98.21 | 100 | |
| Claude 3.7 SonnetAgent Category=Adapted LLM-based Agents2025.05 | 33.33 | 62.5 | 50 | 88.33 | 75.6 | 68.75 |