Step-level tool invocation safety detection on ASB-Traj
0.9497AccuracyTS-Guard
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TS-GuardMode=strict mode2026.01 | 0.9497 | 0.9476 | 0.9385 | |
| TS-GuardMode=Loose Mode2026.01 | 0.9064 | 0.7738 | 0.7824 | |
| TS-GuardEvaluation Mode=Exact Mode2026.01 | 0.8661 | 0.8328 | 0.8322 | |
| gpt-4oMode=Loose Mode2026.01 | 0.8295 | 0.5155 | 0.4435 | |
| Qwen3-8BMode=Loose Mode2026.01 | 0.8148 | 0.4066 | 0.3103 | |
| Qwen3Guard-8B-GenMode=Loose Mode2026.01 | 0.7989 | 0.033 | 0.0168 | |
| Qwen2.5-7B-ITMode=Loose Mode2026.01 | 0.7928 | 0.2386 | 0.1589 | |
| Llama-Guard-3-8BMode=Loose Mode2026.01 | 0.767 | 0.2789 | 0.2203 | |
| SafironMode=Loose Mode2026.01 | 0.7065 | 0.4397 | 0.563 | |
| ShieldAgent-THUMode=Loose Mode2026.01 | 0.6641 | 0.4834 | 0.7684 | |
| GPT-4oMode=strict mode2026.01 | 0.6584 | 0.6303 | 0.6011 | |
| Qwen3-8BMode=strict mode2026.01 | 0.5966 | 0.3893 | 0.2654 | |
| ShieldAgent-THUMode=strict mode2026.01 | 0.5763 | 0.5444 | 0.5227 | |
| Qwen3-8BEvaluation Mode=Exact Mode2026.01 | 0.5494 | 0.4028 | 0.4275 | |
| gpt-4oEvaluation Mode=Exact Mode2026.01 | 0.5492 | 0.5057 | 0.4939 | |
| Llama-Guard-3-8BMode=strict mode2026.01 | 0.5467 | 0.2482 | 0.1545 | |
| Qwen3Guard-8B-GenMode=strict mode2026.01 | 0.535 | 0.1362 | 0.0757 | |
| Qwen2.5-7B-ITMode=strict mode2026.01 | 0.521 | 0.6296 | 0.8407 | |
| SafironMode=strict mode2026.01 | 0.5163 | 0.3982 | 0.3303 | |
| Qwen2.5-7B-ITEvaluation Mode=Exact Mode2026.01 | 0.3585 | 0.3254 | 0.3791 |