Agent Safety Evaluation on AgentDojo held-out
46.8ASRFATE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| FATEBackbone=Gemma-3-12B-it2026.05 | 46.8 | 46.2 | 9.1 | |
| FATEBackbone=Ministral-3-8B-Instruct2026.05 | 48.6 | 43.8 | 7.4 | |
| FATEBackbone=Phi-4-reasoning2026.05 | 50.3 | 42.9 | 8.9 | |
| FATEBackbone=Llama-3.1-8B-Instruct2026.05 | 51.2 | 41.7 | 8.7 | |
| FATEBackbone=Qwen3-8B-Instruct2026.05 | 54 | 39.2 | 8.2 | |
| BaseBackbone=Gemma-3-12B-it2026.05 | 70.4 | 20.4 | 13.2 | |
| BaseBackbone=Ministral-3-8B-Instruct2026.05 | 73.6 | 17.6 | 9.6 | |
| BaseBackbone=Phi-4-reasoning2026.05 | 74.8 | 16.8 | 12.6 | |
| BaseBackbone=Llama-3.1-8B-Instruct2026.05 | 76.8 | 15.8 | 11.8 | |
| BaseBackbone=Qwen3-8B-Instruct2026.05 | 81.2 | 13.2 | 10.4 |