Utility Assessment on AgentDojo (test)
100UtilitySemantic-Injection
Evaluation results
| Method | Links | |
|---|---|---|
| Semantic-InjectionTarget Model=Gemini-3-Flash, Defense=w/o Defense2026.02 | 100 | |
| Semantic-InjectionTarget Model=Gemini-3-Flash, Defense=Delimiter Spotlighting2026.02 | 100 | |
| ChatInjectTarget Model=Gemini-3-Flash, Defense=Delimiter Spotlighting2026.02 | 100 | |
| ChatInjectTarget Model=Gemini-3-Flash, Defense=Tag Filter2026.02 | 100 | |
| ChatInjectTarget Model=Gemini-3-Flash, Defense=w/o Defense2026.02 | 99.35 | |
| Single-TemplateTarget Model=Gemini-3-Flash, Defense=Tag Filter2026.02 | 99.35 | |
| Semantic-InjectionTarget Model=Gemini-3-Flash, Defense=Tag Filter2026.02 | 98.7 | |
| Single-TemplateTarget Model=Gemini-3-Flash, Defense=Delimiter Spotlighting2026.02 | 98.05 | |
| ChatInjectTarget Model=DeepSeek-V3.2, Defense=Delimiter Spotlighting2026.02 | 93.51 | |
| ChatInjectTarget Model=GPT-4.1, Defense=w/o Defense2026.02 | 91.56 | |
| ChatInjectTarget Model=GPT-4.1, Defense=Delimiter Spotlighting2026.02 | 90.26 | |
| ChatInjectTarget Model=GPT-4.1, Defense=Tag Filter2026.02 | 90.26 | |
| ChatInjectTarget Model=DeepSeek-V3.2, Defense=w/o Defense2026.02 | 88.31 | |
| Single-TemplateTarget Model=DeepSeek-V3.2, Defense=Tag Filter2026.02 | 88.31 | |
| ChatInjectTarget Model=DeepSeek-V3.2, Defense=Tag Filter2026.02 | 87.66 | |
| Single-TemplateTarget Model=Gemini-3-Flash, Defense=w/o Defense2026.02 | 87.01 | |
| ChatInjectTarget Model=GPT-4.1-Mini, Defense=Tag Filter2026.02 | 84.42 | |
| ChatInjectTarget Model=GPT-4o, Defense=Tag Filter2026.02 | 82.47 | |
| Semantic-InjectionTarget Model=GPT-4o, Defense=Delimiter Spotlighting2026.02 | 79.87 | |
| Semantic-InjectionTarget Model=GPT-4o, Defense=Tag Filter2026.02 | 79.22 | |
| Semantic-InjectionTarget Model=GPT-4o, Defense=w/o Defense2026.02 | 77.92 | |
| Single-TemplateTarget Model=Qwen3-Max, Defense=Tag Filter2026.02 | 77.92 | |
| Semantic-InjectionTarget Model=GPT-4.1, Defense=Delimiter Spotlighting2026.02 | 77.27 | |
| Single-TemplateTarget Model=GPT-4o, Defense=Tag Filter2026.02 | 76.62 | |
| Semantic-InjectionTarget Model=GPT-4.1, Defense=w/o Defense2026.02 | 75.97 | |
| ChatInjectTarget Model=GPT-4.1-Mini, Defense=Delimiter Spotlighting2026.02 | 75.97 | |
| ChatInjectTarget Model=GPT-4.1-Mini, Defense=w/o Defense2026.02 | 74.68 | |
| Semantic-InjectionTarget Model=GPT-4.1, Defense=Tag Filter2026.02 | 74.03 | |
| PhantomTarget Model=Gemini-3-Flash, Defense=Tag Filter2026.02 | 74.03 | |
| Semantic-InjectionTarget Model=Qwen3-Max, Defense=Delimiter Spotlighting2026.02 | 72.08 | |
| ChatInjectTarget Model=Gemini-2.5-Flash, Defense=Delimiter Spotlighting2026.02 | 70.13 | |
| ChatInjectTarget Model=GPT-4o, Defense=Delimiter Spotlighting2026.02 | 68.83 | |
| Single-TemplateTarget Model=Gemini-2.5-Flash, Defense=Delimiter Spotlighting2026.02 | 68.83 | |
| Single-TemplateTarget Model=Gemini-2.5-Flash, Defense=Tag Filter2026.02 | 66.88 | |
| Single-TemplateTarget Model=GPT-4.1, Defense=Delimiter Spotlighting2026.02 | 66.23 | |
| Semantic-InjectionTarget Model=Qwen3-Max, Defense=Tag Filter2026.02 | 64.94 | |
| Single-TemplateTarget Model=GPT-4.1, Defense=w/o Defense2026.02 | 64.29 | |
| ChatInjectTarget Model=Qwen3-Max, Defense=Tag Filter2026.02 | 64.29 | |
| Semantic-InjectionTarget Model=Gemini-2.5-Flash, Defense=Tag Filter2026.02 | 64.29 | |
| Semantic-InjectionTarget Model=Qwen3-Max, Defense=w/o Defense2026.02 | 63.64 | |
| ChatInjectTarget Model=Gemini-2.5-Flash, Defense=Tag Filter2026.02 | 61.69 | |
| ChatInjectTarget Model=Gemini-2.5-Flash, Defense=w/o Defense2026.02 | 61.04 | |
| Semantic-InjectionTarget Model=GPT-4.1-Mini, Defense=Delimiter Spotlighting2026.02 | 61.04 | |
| Semantic-InjectionTarget Model=GPT-4.1-Mini, Defense=Tag Filter2026.02 | 60.39 | |
| ChatInjectTarget Model=GPT-4o, Defense=w/o Defense2026.02 | 59.74 | |
| Single-TemplateTarget Model=Gemini-2.5-Flash, Defense=w/o Defense2026.02 | 59.74 | |
| Semantic-InjectionTarget Model=Qwen-Flash, Defense=Tag Filter2026.02 | 59.09 | |
| PhantomTarget Model=Qwen3-Max, Defense=Tag Filter2026.02 | 58.44 | |
| Semantic-InjectionTarget Model=GPT-4.1-Mini, Defense=w/o Defense2026.02 | 57.14 | |
| Semantic-InjectionTarget Model=Gemini-2.5-Flash, Defense=w/o Defense2026.02 | 57.14 | |
| Semantic-InjectionTarget Model=Gemini-2.5-Flash, Defense=Delimiter Spotlighting2026.02 | 57.14 | |
| Semantic-InjectionTarget Model=Qwen-Flash, Defense=w/o Defense2026.02 | 56.49 | |
| PhantomTarget Model=Gemini-3-Flash, Defense=Delimiter Spotlighting2026.02 | 55.84 | |
| PhantomTarget Model=DeepSeek-V3.2, Defense=Tag Filter2026.02 | 55.84 | |
| ChatInjectTarget Model=Qwen3-Max, Defense=Delimiter Spotlighting2026.02 | 55.19 | |
| PhantomTarget Model=Gemini-2.5-Flash, Defense=Tag Filter2026.02 | 53.9 | |
| PhantomTarget Model=Gemini-2.5-Flash, Defense=Delimiter Spotlighting2026.02 | 53.25 | |
| Semantic-InjectionTarget Model=Qwen-Flash, Defense=Delimiter Spotlighting2026.02 | 50.65 | |
| Single-TemplateTarget Model=Gemini-3-Flash, Defense=Injection Detector2026.02 | 49.35 | |
| ChatInjectTarget Model=Qwen-Flash, Defense=Tag Filter2026.02 | 48.7 | |
| ChatInjectTarget Model=Qwen3-Max, Defense=w/o Defense2026.02 | 46.75 | |
| Single-TemplateTarget Model=GPT-4.1, Defense=Tag Filter2026.02 | 46.1 | |
| Single-TemplateTarget Model=GPT-4o, Defense=Delimiter Spotlighting2026.02 | 45.45 | |
| Single-TemplateTarget Model=GPT-4o, Defense=w/o Defense2026.02 | 44.16 | |
| PhantomTarget Model=GPT-4o, Defense=Tag Filter2026.02 | 43.51 | |
| Single-TemplateTarget Model=Gemini-2.5-Flash, Defense=Injection Detector2026.02 | 43.51 | |
| ChatInjectTarget Model=Gemini-3-Flash, Defense=Injection Detector2026.02 | 42.86 | |
| Semantic-InjectionTarget Model=DeepSeek-V3.2, Defense=Delimiter Spotlighting2026.02 | 42.21 | |
| Semantic-InjectionTarget Model=Gemini-3-Flash, Defense=Injection Detector2026.02 | 42.21 | |
| Semantic-InjectionTarget Model=DeepSeek-V3.2, Defense=Tag Filter2026.02 | 41.56 | |
| PhantomTarget Model=Gemini-2.5-Flash, Defense=w/o Defense2026.02 | 40.91 | |
| Semantic-InjectionTarget Model=DeepSeek-V3.2, Defense=w/o Defense2026.02 | 40.91 | |
| Semantic-InjectionTarget Model=GPT-4o, Defense=Injection Detector2026.02 | 37.01 | |
| Semantic-InjectionTarget Model=Gemini-2.5-Flash, Defense=Injection Detector2026.02 | 34.42 | |
| Single-TemplateTarget Model=Qwen3-Max, Defense=Delimiter Spotlighting2026.02 | 31.82 | |
| ChatInjectTarget Model=Qwen-Flash, Defense=w/o Defense2026.02 | 29.87 | |
| ChatInjectTarget Model=GPT-4.1, Defense=Injection Detector2026.02 | 29.22 | |
| Single-TemplateTarget Model=Qwen3-Max, Defense=w/o Defense2026.02 | 28.57 | |
| PhantomTarget Model=Gemini-2.5-Flash, Defense=Injection Detector2026.02 | 28.57 | |
| PhantomTarget Model=GPT-4o, Defense=Delimiter Spotlighting2026.02 | 27.92 | |
| PhantomTarget Model=Gemini-3-Flash, Defense=w/o Defense2026.02 | 26.62 | |
| Semantic-InjectionTarget Model=Qwen3-Max, Defense=Injection Detector2026.02 | 25.97 | |
| ChatInjectTarget Model=Gemini-2.5-Flash, Defense=Injection Detector2026.02 | 25.97 | |
| ChatInjectTarget Model=Qwen-Flash, Defense=Delimiter Spotlighting2026.02 | 25.32 | |
| Semantic-InjectionTarget Model=GPT-4.1, Defense=Injection Detector2026.02 | 25.32 | |
| PhantomTarget Model=GPT-4o, Defense=w/o Defense2026.02 | 24.03 | |
| Semantic-InjectionTarget Model=Qwen-Flash, Defense=Injection Detector2026.02 | 24.03 | |
| ChatInjectTarget Model=DeepSeek-V3.2, Defense=Injection Detector2026.02 | 24.03 | |
| Single-TemplateTarget Model=GPT-4.1-Mini, Defense=w/o Defense2026.02 | 23.38 | |
| Single-TemplateTarget Model=DeepSeek-V3.2, Defense=w/o Defense2026.02 | 23.38 | |
| Single-TemplateTarget Model=Qwen-Flash, Defense=Tag Filter2026.02 | 23.38 | |
| ChatInjectTarget Model=GPT-4.1-Mini, Defense=Injection Detector2026.02 | 23.38 | |
| PhantomTarget Model=Qwen3-Max, Defense=w/o Defense2026.02 | 22.73 | |
| PhantomTarget Model=DeepSeek-V3.2, Defense=w/o Defense2026.02 | 22.73 | |
| PhantomTarget Model=Qwen3-Max, Defense=Delimiter Spotlighting2026.02 | 22.73 | |
| Semantic-InjectionTarget Model=GPT-4.1-Mini, Defense=Injection Detector2026.02 | 22.73 | |
| Single-TemplateTarget Model=GPT-4.1-Mini, Defense=Delimiter Spotlighting2026.02 | 22.08 | |
| Single-TemplateTarget Model=GPT-4.1, Defense=Injection Detector2026.02 | 22.08 | |
| Semantic-InjectionTarget Model=DeepSeek-V3.2, Defense=Injection Detector2026.02 | 21.43 | |
| PhantomTarget Model=GPT-4.1, Defense=Delimiter Spotlighting2026.02 | 20.13 |