Malicious Tool Call Detection on TRUST-BENCH reconstructed comparison set curated 1,970-episode (test)
100AMRToolShield
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ToolShield2026.05 | 100 | 0 | 0 | 100 | 0 | |
| GuardAgent2026.05 | 100 | 0 | 0 | 100 | 0 | |
| LLM-as-Judge2026.05 | 100 | 0 | 0 | 100 | 0 | |
| Keyword Heuristic2026.05 | 93.3 | 6.7 | 1.5 | 98.5 | 0 | |
| Gradient Boosting (10 scalar feat.)Features=10 scalar features, Model=Gradient Boosting2026.05 | 43.8 | 56.2 | 36.3 | 63.7 | 0 | |
| GPT-4o Judge (features)Model=GPT-4o, Mode=features2026.05 | 25.2 | 74.8 | 80.9 | 19.1 | 0 | |
| GPT-4o Judge (full)Model=GPT-4o, Mode=full2026.05 | 22.2 | 77.8 | 80.4 | 19.6 | 0 | |
| TF-IDF + MLP (same input)Input Representation=Identical structured input, Model=MLP2026.05 | 14.2 | 85.8 | 25.2 | 74.8 | 33.9 | |
| TF-IDF + MLP (raw text)Input Representation=Raw text serialization, Model=MLP2026.05 | 12.2 | 87.8 | 23.8 | 76.2 | 39.9 | |
| TF-IDF + SVM (same input)Input Representation=Identical structured input, Model=SVM2026.05 | 11.7 | 88.3 | 23.2 | 76.8 | 41.7 | |
| TF-IDF + LogReg (raw text)Input Representation=Raw text serialization, Model=Logistic Regression2026.05 | 10.1 | 89.9 | 23.4 | 76.6 | 44.6 | |
| VISTA-Guard (Qwen2.5-7B)Backbone=Qwen2.5-7B, Input Representation=Full structured sequence2026.05 | 9.6 | 90.4 | 10.6 | 89.4 | 64.9 | |
| TF-IDF + LogReg (same input)Input Representation=Identical structured input, Model=Logistic Regression2026.05 | 9.1 | 90.9 | 17.7 | 82.3 | 55.2 | |
| BERT-base (same input)Input Representation=Condensed structured input (512 tokens), Backbone=BERT-base2026.05 | 8 | 92 | 48.1 | 51.9 | 21.9 | |
| VISTA-Guard (Qwen2-8B)Backbone=Qwen2-8B, Input Representation=Full structured sequence2026.05 | 7.6 | 92.4 | 11.6 | 88.4 | 67.6 | |
| VISTA-Guard (OLMo-2-7B)Backbone=OLMo-2-7B, Input Representation=Full structured sequence2026.05 | 6.6 | 93.4 | 15.1 | 84.9 | 64.2 | |
| TF-IDF + SVM (raw text)Input Representation=Raw text serialization, Model=SVM2026.05 | 6.1 | 93.9 | 30.9 | 69.1 | 41.5 | |
| VISTA-Guard (Meta CWM-32B)Backbone=Meta CWM-32B, Input Representation=Full structured sequence2026.05 | 5.2 | 94.8 | 40.4 | 59.6 | 39 | |
| VISTA-Guard (Llama-3.1-8B)Backbone=Llama-3.1-8B, Input Representation=Full structured sequence2026.05 | 4.6 | 95.4 | 18.8 | 81.2 | 62.6 | |
| VISTA-Guard (Mistral-7B)Backbone=Mistral-7B, Input Representation=Full structured sequence2026.05 | 4.2 | 95.8 | 5 | 95 | 84.2 |