Step-level tool invocation safety detection on AgentHarm Traj
84.81AccuracyTS-Guard
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TS-GuardMode=strict mode2026.01 | 84.81 | 90.16 | 96.95 | |
| Llama-Guard-3-8BMode=strict mode2026.01 | 81.53 | 86.35 | 81.33 | |
| Qwen3Guard-8B-GenMode=strict mode2026.01 | 80.57 | 86.27 | 84.95 | |
| TS-GuardEvaluation Mode=Exact Mode2026.01 | 79.93 | 75.93 | 75.99 | |
| gpt-4oMode=Loose Mode2026.01 | 79.48 | 63.94 | 63.33 | |
| Qwen3-8BMode=Loose Mode2026.01 | 76.47 | 47.24 | 36.67 | |
| GPT-4oMode=strict mode2026.01 | 75.23 | 84.8 | 96.19 | |
| Qwen2.5-7B-ITMode=Loose Mode2026.01 | 74.28 | 29.32 | 18.57 | |
| ShieldAgent-THUMode=strict mode2026.01 | 71.13 | 82.63 | 95.62 | |
| Qwen3Guard-8B-GenMode=Loose Mode2026.01 | 70.72 | 47.8 | 46.67 | |
| Qwen2.5-7B-ITMode=strict mode2026.01 | 67.99 | 80.17 | 90.09 | |
| TS-GuardMode=Loose Mode2026.01 | 64.84 | 55.92 | 77.62 | |
| SafironMode=Loose Mode2026.01 | 64.02 | 36.32 | 35.71 | |
| gpt-4oEvaluation Mode=Exact Mode2026.01 | 58.14 | 53.63 | 54.53 | |
| Llama-Guard-3-8BMode=Loose Mode2026.01 | 56.77 | 53.11 | 85.24 | |
| Qwen3-8BMode=strict mode2026.01 | 54.46 | 58.94 | 45.52 | |
| SafironMode=strict mode2026.01 | 46.1 | 45.88 | 31.81 | |
| Qwen3-8BEvaluation Mode=Exact Mode2026.01 | 45.14 | 44.95 | 47.9 | |
| Qwen2.5-7B-ITEvaluation Mode=Exact Mode2026.01 | 45 | 34.85 | 38.22 | |
| ShieldAgent-THUMode=Loose Mode2026.01 | 33.24 | 45.78 | 98.09 |