Step-level tool invocation safety detection on AgentDojo-Traj
91.72AccuracyTS-Guard
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TS-GuardMode=strict mode2026.01 | 91.72 | 86.18 | 89.49 | |
| TS-GuardMode=Loose Mode2026.01 | 85.81 | 68.26 | 52.84 | |
| Qwen3-8BMode=Loose Mode2026.01 | 85.47 | 77.1 | 84.9 | |
| gpt-4oMode=Loose Mode2026.01 | 84.92 | 79.14 | 99.15 | |
| TS-GuardEvaluation Mode=Exact Mode2026.01 | 81.15 | 54.11 | 50.48 | |
| Qwen3-8BMode=strict mode2026.01 | 79.65 | 72.32 | 92.31 | |
| Qwen3-8BEvaluation Mode=Exact Mode2026.01 | 77.52 | 53.67 | 53.15 | |
| Llama-Guard-3-8BMode=strict mode2026.01 | 74.75 | 33.33 | 21.87 | |
| Llama-Guard-3-8BMode=Loose Mode2026.01 | 74.75 | 33.33 | 21.87 | |
| Qwen2.5-7B-ITMode=Loose Mode2026.01 | 74.73 | 50.95 | 45.58 | |
| Qwen3Guard-8B-GenMode=Loose Mode2026.01 | 71.15 | 0 | 0 | |
| SafironMode=strict mode2026.01 | 70.98 | 52.8 | 56.25 | |
| SafironMode=Loose Mode2026.01 | 70.98 | 52.8 | 56.25 | |
| Qwen3Guard-8B-GenMode=strict mode2026.01 | 70.57 | 3.23 | 1.7 | |
| ShieldAgent-THUMode=strict mode2026.01 | 60.9 | 58.91 | 97.16 | |
| ShieldAgent-THUMode=Loose Mode2026.01 | 60.9 | 58.91 | 97.16 | |
| GPT-4oMode=strict mode2026.01 | 55.74 | 56.59 | 100 | |
| gpt-4oEvaluation Mode=Exact Mode2026.01 | 55.49 | 44.66 | 45.64 | |
| Qwen2.5-7B-ITMode=strict mode2026.01 | 43.97 | 50.47 | 99.14 | |
| Qwen2.5-7B-ITEvaluation Mode=Exact Mode2026.01 | 28.55 | 28.82 | 22.41 |