Unauthorized Tool Call Attempt Detection on Task-aligned prompt injection sweep
0ASR (Task-aligned)gemini-3.1-pro-preview
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| gemini-3.1-pro-previewEvidence class=Tier sweep, CI / significance=Flagship low-ASR exemplar.2026.06 | 0 | — | — | — | |
| Latam-GPT ESEvidence class=Sovereign model, Model / split=bnb-4bit text-tool, CI / significance=Paired CI [-0.327, -0.072]; McNemar p = 0.004; nelig = 60.2026.06 | 4 | 24 | 20 | 20 | |
| glm-5.2Evidence class=Tier sweep, CI / significance=Flagship low-ASR exemplar.2026.06 | 8.7 | — | — | — | |
| Flagship meanEvidence class=Tier sweep mean, Model / split=15 models, CI / significance=Aggregate sweep; high outliers present.2026.06 | 18.9 | — | — | — | |
| mistral-largeEvidence class=Paired ∆, CI / significance=∆ CI [0.311, 0.545]; BH q < 0.001; n = 74.2026.06 | 45.9 | 2.7 | 43.2 | 43.9 | |
| Latam-GPT ENEvidence class=Sovereign model, Model / split=bnb-4bit text-tool, CI / significance=Paired CI [-0.007, 0.275]; McNemar p = 0.096; nelig = 58.2026.06 | 52 | 32 | 20 | 13.8 | |
| gemma-3-12b-itEvidence class=Paired ∆, CI / significance=∆ CI [0.444, 0.681]; BH q < 0.001; n = 70.2026.06 | 57.1 | 0 | 57.1 | 57.1 | |
| Deployment-tier meanEvidence class=Tier sweep mean, Model / split=12 models, CI / significance=Aggregate sweep; no paired CI in source.2026.06 | 60.3 | — | — | — | |
| gemini-3.1-flash-liteEvidence class=Tier sweep, CI / significance=Deployment-tier exemplar.2026.06 | 98 | — | — | — | |
| gpt-4o-miniEvidence class=Tier sweep, CI / significance=Deployment-tier exemplar.2026.06 | 100 | — | — | — | |
| glm-4.5-airEvidence class=Tier sweep, CI / significance=Deployment-tier exemplar.2026.06 | 100 | — | — | — | |
| glm-4.7Evidence class=Paired ∆, CI / significance=∆ CI [0.555, 0.827]; BH q < 0.001; n = 46.2026.06 | 100 | 28.3 | 71.7 | 87 |