Agent Safety on AuraGen
95.31AccuracyExtractor
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ExtractorBackbone=Llama-3.1-8B-Instruct2026.02 | 95.31 | 94.06 | 98.96 | 89.62 | |
| DRAFT (Llama-3.1-8B)Backbone=Llama-3.1-8B, Adaptation=DRAFT2026.02 | 94.01 | 92.13 | — | 88.79 | |
| DRAFT (Qwen3-4B-Instruct-2507)Backbone=Qwen3-4B-Instruct-2507, Adaptation=DRAFT2026.02 | 93.88 | 92.23 | — | 91.13 | |
| DRAFT (Qwen3Guard-Gen-4B)Backbone=Qwen3Guard-Gen-4B, Adaptation=DRAFT2026.02 | 93.62 | 91.55 | — | 88.59 | |
| DRAFTBackbone=Llama-3.1-8B-Instruct2026.02 | 93.36 | 91.79 | 94.06 | 89.62 | |
| DRAFTBackbone=Qwen3-4B2026.02 | 92.96 | 91.79 | 94.06 | 88.53 | |
| DRAFT (Qwen3-8B)Backbone=Qwen3-8B, Adaptation=DRAFT2026.02 | 92.06 | 89.69 | — | 89.27 | |
| DRAFTBackbone=Qwen2.5-7B-Instruct2026.02 | 90.62 | 88.35 | 91 | 85.85 | |
| Qwen3-8B (SFT)Backbone=Qwen3-8B, Adaptation=SFT2026.02 | 90.49 | 87.29 | — | 83.06 | |
| Qwen3-4B-Instruct-2507 (SFT)Backbone=Qwen3-4B-Instruct-2507, Adaptation=SFT2026.02 | 86.91 | 83.81 | — | 81.07 | |
| SFTBackbone=Qwen3-4B2026.02 | 85.55 | 80 | 93.67 | 69.81 | |
| SFTBackbone=Llama-3.1-8B-Instruct2026.02 | 84.77 | 78.92 | 92.41 | 68.87 | |
| ExtractorBackbone=Qwen3-4B2026.02 | 80.08 | 71.82 | 86.67 | 61.32 | |
| Llama-3.1-8B (SFT)Backbone=Llama-3.1-8B, Adaptation=SFT2026.02 | 79.39 | 64.96 | — | 56.99 | |
| Qwen3Guard-Gen-4B (SFT)Backbone=Qwen3Guard-Gen-4B, Adaptation=SFT2026.02 | 79.03 | 56.32 | — | 50.38 | |
| ExtractorBackbone=Qwen2.5-1.5B-Instruct2026.02 | 78.12 | 66.27 | 91.67 | 51.89 | |
| DRAFTBackbone=Qwen2.5-3B-Instruct2026.02 | 70.7 | 63.05 | 65.98 | 60.38 | |
| Llama-3.1-8B (AgentAuditor)Backbone=Llama-3.1-8B, Adaptation=AA2026.02 | 70.42 | 68.42 | — | 67.25 | |
| Qwen3-8B (AgentAuditor)Backbone=Qwen3-8B, Adaptation=AA2026.02 | 69.84 | 40.79 | — | 29.25 | |
| Qwen3-4B-Instruct-2507 (LoRA)Backbone=Qwen3-4B-Instruct-2507, Adaptation=LoRA2026.02 | 69.17 | 71.76 | — | 66.2 | |
| Qwen3-8B (LoRA)Backbone=Qwen3-8B, Adaptation=LoRA2026.02 | 64.38 | 17.14 | — | 13.77 | |
| ExtractorBackbone=Qwen2.5-3B-Instruct2026.02 | 63.28 | 26.56 | 77.27 | 16.04 | |
| VanillaBackbone=Qwen2.5-7B-Instruct2026.02 | 63.21 | 30.36 | 58.62 | 20.48 | |
| VanillaBackbone=Llama-3.1-8B-Instruct2026.02 | 62.74 | 8.14 | 87.5 | 4.27 | |
| Llama-3.1-8B (Vanilla)Backbone=Llama-3.1-8B, Adaptation=Vanilla2026.02 | 62.2 | 16.84 | — | 10.54 | |
| Llama-3.1-8B (LoRA)Backbone=Llama-3.1-8B, Adaptation=LoRA2026.02 | 62.11 | 19.83 | — | 11.32 | |
| ExtractorBackbone=Qwen2.5-7B-Instruct2026.02 | 61.91 | 20.63 | 65 | 12.26 | |
| LoRABackbone=Qwen3-4B2026.02 | 61.72 | 18.33 | 78.57 | 10.38 | |
| VanillaBackbone=Qwen2.5-1.5B-Instruct2026.02 | 61.56 | 0 | 0 | 0 | |
| LoRABackbone=Qwen2.5-7B-Instruct2026.02 | 60.94 | 27.54 | 59.38 | 17.92 | |
| Qwen3-4B-Instruct-2507 (AgentAuditor)Backbone=Qwen3-4B-Instruct-2507, Adaptation=AA2026.02 | 60.56 | 53.87 | — | 45.45 | |
| Qwen3-8B (Vanilla)Backbone=Qwen3-8B, Adaptation=Vanilla2026.02 | 60.53 | 15.44 | — | 12.83 | |
| Qwen3Guard-Gen-4B (Vanilla)Backbone=Qwen3Guard-Gen-4B, Adaptation=Vanilla2026.02 | 60.47 | 10.51 | — | 13.43 | |
| Qwen3Guard-Gen-4B (LoRA)Backbone=Qwen3Guard-Gen-4B, Adaptation=LoRA2026.02 | 60.16 | 10.53 | — | 5.66 | |
| VanillaBackbone=Qwen2.5-3B-Instruct2026.02 | 60.14 | 19.91 | 51.22 | 12.35 | |
| Qwen3Guard-Gen-4B (AgentAuditor)Backbone=Qwen3Guard-Gen-4B, Adaptation=AA2026.02 | 59.77 | 52.97 | — | 41.6 | |
| Qwen3-4B-Instruct-2507 (Vanilla)Backbone=Qwen3-4B-Instruct-2507, Adaptation=Vanilla2026.02 | 59.7 | 53.39 | — | 58.96 | |
| SFTBackbone=Qwen2.5-1.5B-Instruct2026.02 | 58.59 | 0 | 0 | 0 | |
| DRAFTBackbone=Qwen2.5-1.5B-Instruct2026.02 | 58.59 | 25.01 | 18.36 | 13.87 | |
| SFTBackbone=Qwen2.5-3B-Instruct2026.02 | 58.59 | 7.02 | 50 | 3.77 | |
| VanillaBackbone=Qwen3-4B2026.02 | 58.25 | 13.66 | 70 | 7.57 | |
| LoRABackbone=Qwen2.5-1.5B-Instruct2026.02 | 58.2 | 0 | 0 | 0 | |
| ChatGPT 5.2 (API)Backbone=ChatGPT 5.2, Adaptation=API2026.02 | 58.01 | 55.79 | — | 44.26 | |
| SFTBackbone=Qwen2.5-7B-Instruct2026.02 | 57.81 | 1.82 | 25 | 0.94 | |
| LoRABackbone=Llama-3.1-8B-Instruct2026.02 | 57.81 | 3.57 | 33.33 | 1.89 | |
| LoRABackbone=Qwen2.5-3B-Instruct2026.02 | 57.42 | 18.05 | 44.44 | 11.32 | |
| gpt-oss-120b (Vanilla)Backbone=gpt-oss-120b, Adaptation=Vanilla2026.02 | 53.92 | 51.76 | — | 39.22 |