Tool-use security and utility evaluation on Security
98.4Utility Rategpt-5.4-mini
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| gpt-5.4-miniTier=frontier, Mode=aware2026.05 | 98.4 | 0.5 | 0.4 | 0 | 89.9 | |
| Gemma-4-26B-A4B-itTier=open, Mode=aware2026.05 | 95.3 | 3.6 | 0.4 | 66.7 | 73.6 | |
| Claude Sonnet 4.6Tier=frontier, Mode=aware2026.05 | 78.5 | 5.2 | 1.6 | 66.7 | 77.8 | |
| Gemma-4-26B-A4B-itTier=open, Mode=blind2026.05 | 72.9 | 6.2 | 0 | 0 | 57 | |
| gpt-oss-20BTier=open, Mode=aware2026.05 | 71.4 | 26.6 | 0 | 100 | 54.5 | |
| Gemini 3.1 Flash LiteTier=frontier, Mode=aware2026.05 | 69.8 | 14.8 | 0 | 100 | 59.6 | |
| gpt-oss-20BTier=open, Mode=blind2026.05 | 60.3 | 2.5 | 0 | 0 | 46.1 | |
| Ministral-3-14BTier=open, Mode=blind2026.05 | 57.8 | 4.2 | 0 | 0 | 50.4 | |
| Gemini 3.1 Flash LiteTier=frontier, Mode=blind2026.05 | 53.2 | 32.1 | 0 | 100 | 46.5 | |
| Qwen3.6-35B-A3BTier=open, Mode=blind2026.05 | 48.4 | 40.1 | 1.5 | 66.7 | 42.2 | |
| Qwen3.6-35B-A3BTier=open, Mode=aware2026.05 | 40.1 | 58.3 | 1.2 | 66.7 | 37.6 | |
| gpt-5.4-miniTier=frontier, Mode=blind2026.05 | 28.1 | 54.2 | 0.8 | 100 | 32.2 | |
| Claude Haiku 4.5Tier=frontier, Mode=aware2026.05 | 27.9 | 67.9 | 0 | 100 | 39.5 | |
| Claude Sonnet 4.6Tier=frontier, Mode=blind2026.05 | 11.2 | 77.1 | 11.9 | 100 | 18.6 | |
| Ministral-3-14BTier=open, Mode=aware2026.05 | 7.8 | 88.5 | 0 | 100 | 14.3 | |
| Claude Haiku 4.5Tier=frontier, Mode=blind2026.05 | 6.6 | 84.1 | 2.8 | 100 | 14.9 |