Agent Safety on R-Judge
97.3AccuracyGemini-3.1-Pro
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Gemini-3.1-Pro2026.05 | 97.3 | 97.4 | 99.1 | 95.7 | — | |
| Gemini-3-Flash2026.05 | 95.2 | 95.3 | 98.7 | 92.1 | — | |
| SFTBackbone=Llama-3.1-8B-Instruct2026.02 | 94.25 | 94.62 | 91.67 | 94.58 | — | |
| DRAFT (Qwen3-8B)Backbone=Qwen3-8B, Adaptation=DRAFT2026.02 | 93.4 | 92.13 | — | 92.49 | — | |
| TRACEBackbone=Qwen3-8B2026.05 | 93.4 | 92.13 | — | — | 92.49 | |
| GPT-5.42026.05 | 93.3 | 93.7 | 93.1 | 94.3 | — | |
| DRAFTBackbone=Llama-3.1-8B-Instruct2026.02 | 93.27 | 93.48 | 91.49 | 94.77 | — | |
| SFTBackbone=Qwen2.5-3B-Instruct2026.02 | 93.1 | 93.75 | 88.24 | 97.16 | — | |
| ExtractorBackbone=Llama-3.1-8B-Instruct2026.02 | 93.1 | 93.62 | 89.8 | 95.67 | — | |
| MAGEBackbone=Qwen3-8B2026.05 | 92.56 | 78.43 | — | — | 78.91 | |
| Qwen3-8B (SFT)Backbone=Qwen3-8B, Adaptation=SFT2026.02 | 92.39 | 92.91 | — | 92.26 | — | |
| SFTBackbone=Qwen3-8B2026.05 | 92.39 | 92.91 | — | — | 92.26 | |
| AgentDoG 1.5-4BBase Model=Qwen3.5 Base2026.05 | 92.2 | 92.7 | 91.7 | 93.7 | — | |
| DRAFT (Llama-3.1-8B)Backbone=Llama-3.1-8B, Adaptation=DRAFT2026.02 | 92.07 | 92.66 | — | 98.12 | — | |
| TRACEBackbone=Llama-3.1-8B2026.05 | 92.07 | 92.66 | — | — | 98.12 | |
| DRAFT (Qwen3Guard-Gen-4B)Backbone=Qwen3Guard-Gen-4B, Adaptation=DRAFT2026.02 | 92.01 | 93.12 | — | 92.17 | — | |
| TRACEBackbone=Qwen3Guard-Gen-4B2026.05 | 92.01 | 93.12 | — | — | 92.17 | |
| ExtractorBackbone=Qwen2.5-3B-Instruct2026.02 | 91.95 | 92.63 | 88.07 | 97.78 | — | |
| AgentDoG 1.0-4BBase Model=Qwen3 Base2026.05 | 91.8 | 92.7 | 87.5 | 98.5 | — | |
| MAGEBackbone=Llama-3.1-8B2026.05 | 91.57 | 80.45 | — | — | 82.78 | |
| DRAFT (Qwen3-4B-Instruct-2507)Backbone=Qwen3-4B-Instruct-2507, Adaptation=DRAFT2026.02 | 91.39 | 91.84 | — | 91.26 | — | |
| TRACEBackbone=Qwen3-4B-Instruct-25072026.05 | 91.39 | 91.84 | — | — | 91.26 | |
| Llama-3.1-8B (SFT)Backbone=Llama-3.1-8B, Adaptation=SFT2026.02 | 91.24 | 92.37 | — | 97.87 | — | |
| SFTBackbone=Llama-3.1-8B2026.05 | 91.24 | 92.37 | — | — | 97.87 | |
| Qwen3Guard-Gen-4B (SFT)Backbone=Qwen3Guard-Gen-4B, Adaptation=SFT2026.02 | 91.14 | 93.55 | — | 93.52 | — | |
| SFTBackbone=Qwen3Guard-Gen-4B2026.05 | 91.14 | 93.55 | — | — | 93.52 | |
| SFTBackbone=Qwen2.5-7B-Instruct2026.02 | 90.8 | 91.31 | 89.36 | 93.34 | — | |
| GPT-5.22026.05 | 90.8 | 91.8 | 86.8 | 97.5 | — | |
| AgentDoG 1.5-4B-UBase Model=Qwen3.5 Base2026.05 | 90.4 | 90.6 | 93.9 | 87.6 | — | |
| MAGEBackbone=Qwen3-4B-Instruct-25072026.05 | 89.67 | 89.43 | — | — | 75.21 | |
| SFTBackbone=Qwen2.5-1.5B-Instruct2026.02 | 89.66 | 90.53 | 86 | 95.56 | — | |
| QwQ-32B2026.05 | 89.5 | 89.5 | 94.9 | 84.7 | — | |
| Qwen3-4B-Instruct-2507 (SFT)Backbone=Qwen3-4B-Instruct-2507, Adaptation=SFT2026.02 | 88.51 | 89.36 | — | 93.34 | — | |
| SFTBackbone=Qwen3-4B-Instruct-25072026.05 | 88.51 | 89.36 | — | — | 93.34 | |
| DRAFTBackbone=Qwen3-4B2026.02 | 87.36 | 87.91 | 96.96 | 88.89 | — | |
| ExtractorBackbone=Qwen2.5-1.5B-Instruct2026.02 | 86.21 | 86.96 | 85.11 | 88.89 | — | |
| SFTBackbone=Qwen3-4B2026.02 | 86.21 | 88.24 | 78.95 | 96.04 | — | |
| DRAFTBackbone=Qwen2.5-3B-Instruct2026.02 | 86.05 | 87.23 | 82.14 | 93.18 | — | |
| Qwen3.5-397B-A17B2026.05 | 85.6 | 87.4 | 81.3 | 94.5 | — | |
| Qwen3-235B-A22B-Instruct-25072026.05 | 85.1 | 87 | 80.7 | 94.4 | — | |
| ExtractorBackbone=Qwen3-4B2026.02 | 85.06 | 85.06 | 88.1 | 82.42 | — | |
| Qwen3.5-4B2026.05 | 81 | 82 | 82.1 | 81.9 | — | |
| ShieldAgent2026.05 | 81 | 84.6 | 74 | 98.8 | — | |
| DRAFTBackbone=Qwen2.5-7B-Instruct2026.02 | 80.46 | 82.83 | 75.93 | 92.01 | — | |
| MAGEBackbone=Qwen3Guard-Gen-4B2026.05 | 80.34 | 77.69 | — | — | 76.21 | |
| ChatGPT 5.2 (API)Backbone=ChatGPT 5.2, Adaptation=API2026.02 | 79.59 | 74.07 | — | 61.37 | — | |
| DRAFTBackbone=Qwen2.5-1.5B-Instruct2026.02 | 78.97 | 75.82 | 79.47 | 77.78 | — | |
| Qwen3-8B (AgentAuditor)Backbone=Qwen3-8B, Adaptation=AA2026.02 | 78.64 | 70.57 | — | 72.21 | — | |
| AABackbone=Qwen3-8B2026.05 | 78.64 | 70.57 | — | — | 72.21 | |
| AgentDoG 1.5-0.8BBase Model=Qwen3.5 Base2026.05 | 75.7 | 74.6 | 83.3 | 67.5 | — | |
| AgentDoG 1.5-8BBase Model=Llama-3.1 Base2026.05 | 75.5 | 81 | 68.6 | 98.8 | — | |
| Llama-3.1-8B (AgentAuditor)Backbone=Llama-3.1-8B, Adaptation=AA2026.02 | 75.33 | 76.96 | — | 78.89 | — | |
| AABackbone=Llama-3.1-8B2026.05 | 75.33 | 76.96 | — | — | 78.89 | |
| Qwen3Guard-Gen-4B (AgentAuditor)Backbone=Qwen3Guard-Gen-4B, Adaptation=AA2026.02 | 75.21 | 72.96 | — | 71.89 | — | |
| AABackbone=Qwen3Guard-Gen-4B2026.05 | 75.21 | 72.96 | — | — | 71.89 | |
| ExtractorBackbone=Qwen2.5-7B-Instruct2026.02 | 72.41 | 77.36 | 67.21 | 91.26 | — | |
| AgentDoG 1.5-2BBase Model=Qwen3.5 Base2026.05 | 71.5 | 70.4 | 78 | 64.1 | — | |
| Qwen3-4B-Instruct-2507 (LoRA)Backbone=Qwen3-4B-Instruct-2507, Adaptation=LoRA2026.02 | 68.97 | 52.73 | — | 59.08 | — | |
| Qwen3-4B-Instruct-25072026.05 | 68.4 | 67.6 | 73.8 | 62.4 | — | |
| Qwen2.5-7B-Instruct2026.05 | 68.4 | 65.5 | 77.4 | 56.8 | — | |
| Qwen3-4B-Instruct-2507 (AgentAuditor)Backbone=Qwen3-4B-Instruct-2507, Adaptation=AA2026.02 | 67.82 | 55.2 | — | 53.49 | — | |
| AABackbone=Qwen3-4B-Instruct-25072026.05 | 67.82 | 55.2 | — | — | 53.49 | |
| gpt-oss-120b (Vanilla)Backbone=gpt-oss-120b, Adaptation=Vanilla2026.02 | 67.69 | 64.42 | — | 55.36 | — | |
| LlamaGuard4-12B2026.05 | 63.8 | 63.2 | 68.3 | 58.8 | — | |
| VanillaBackbone=Llama-3.1-8B-Instruct2026.02 | 61.92 | 49.32 | 81.82 | 35.29 | — | |
| LlamaGuard3-8B2026.05 | 61.2 | 56.7 | 69.1 | 48.1 | — | |
| LoRABackbone=Qwen2.5-3B-Instruct2026.02 | 57.47 | 53.16 | 61.76 | 46.67 | — | |
| LoRABackbone=Llama-3.1-8B-Instruct2026.02 | 57.47 | 44.78 | 68.18 | 33.33 | — | |
| VanillaBackbone=Qwen2.5-7B-Instruct2026.02 | 57.12 | 65.47 | 56.7 | 77.45 | — | |
| VanillaBackbone=Qwen2.5-3B-Instruct2026.02 | 56.76 | 50.49 | 62.93 | 42.16 | — | |
| LoRABackbone=Qwen2.5-7B-Instruct2026.02 | 56.32 | 64.81 | 55.56 | 77.78 | — | |
| VanillaBackbone=Qwen3-4B2026.02 | 55.17 | 36.95 | 62.99 | 26.14 | — | |
| NemoGuard2026.05 | 54.4 | 48.5 | 60.1 | 40.6 | — | |
| Qwen3.5-2B2026.05 | 54.1 | 36.7 | 67.6 | 25.2 | — | |
| Llama-3.1-8B-Instruct2026.05 | 53.7 | 69.5 | 53.3 | 99.8 | — | |
| Qwen3-4B-Instruct-2507 (Vanilla)Backbone=Qwen3-4B-Instruct-2507, Adaptation=Vanilla2026.02 | 53.46 | 59.18 | — | 58.1 | — | |
| BaseBackbone=Qwen3-4B-Instruct-25072026.05 | 53.46 | 59.18 | — | — | 58.1 | |
| Qwen3Guard-Gen-4B (LoRA)Backbone=Qwen3Guard-Gen-4B, Adaptation=LoRA2026.02 | 52.87 | 28.07 | — | 17.78 | — | |
| JoySafety2026.05 | 52.5 | 47.2 | 57.2 | 40.2 | — | |
| Qwen3Guard-Gen-4B (Vanilla)Backbone=Qwen3Guard-Gen-4B, Adaptation=Vanilla2026.02 | 51.8 | 26.25 | — | 16.34 | — | |
| BaseBackbone=Qwen3Guard-Gen-4B2026.05 | 51.8 | 26.25 | — | — | 16.34 | |
| LoRABackbone=Qwen3-4B2026.02 | 51.72 | 32.26 | 58.82 | 25.74 | — | |
| Llama-3.1-8B (LoRA)Backbone=Llama-3.1-8B, Adaptation=LoRA2026.02 | 48.28 | 14.26 | — | 12.22 | — | |
| Qwen3-8B (LoRA)Backbone=Qwen3-8B, Adaptation=LoRA2026.02 | 47.93 | 15.62 | — | 12.11 | — | |
| Llama-3.1-8B (Vanilla)Backbone=Llama-3.1-8B, Adaptation=Vanilla2026.02 | 46.31 | 16.63 | — | 17.33 | — | |
| BaseBackbone=Llama-3.1-8B2026.05 | 46.31 | 16.63 | — | — | 17.33 | |
| VanillaBackbone=Qwen2.5-1.5B-Instruct2026.02 | 45.45 | 43.01 | 40.16 | 37.66 | — | |
| LoRABackbone=Qwen2.5-1.5B-Instruct2026.02 | 44.83 | 41.37 | 39.58 | 37.36 | — | |
| Qwen3-8B (Vanilla)Backbone=Qwen3-8B, Adaptation=Vanilla2026.02 | 41.85 | 20.61 | — | 14.38 | — | |
| BaseBackbone=Qwen3-8B2026.05 | 41.85 | 20.61 | — | — | 14.38 | |
| Qwen3-Guard2026.05 | 40.6 | 9 | 23.6 | 5.6 | — | |
| Qwen3.5-0.8B2026.05 | 33.7 | 20.1 | 27.6 | 15.8 | — |