Policy Violation Localization on AgentDojo, MSB, and AgentDAM Average
88AUROC @5GIF
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GIFAgent Model=GPT-A2026.06 | 88 | 94 | 95 | 96 | 97 | 85 | 93 | 94 | 95 | 96 | 16.38 | |
| GPT (Full Trajectory Judge)Agent Model=GPT-A2026.06 | 86 | 88 | 88 | 95 | 96 | 82 | 85 | 85 | 93 | 94 | 160.91 | |
| GIFAgent Model=Qwen-C2026.06 | 84 | 90 | 91 | 91 | 93 | 76 | 80 | 82 | 82 | 84 | 3.92 | |
| GIFAgent Model=Average2026.06 | 80 | 87 | 88 | 90 | 91 | 70 | 77 | 80 | 82 | 83 | 15.98 | |
| GPT (Full Trajectory Judge)Agent Model=GPT-B2026.06 | 78 | 76 | 78 | 94 | 96 | 62 | 60 | 62 | 84 | 88 | 103.37 | |
| GIFAgent Model=Gemma42026.06 | 78 | 81 | 80 | 83 | 83 | 57 | 60 | 60 | 62 | 63 | 36.25 | |
| GPT (Full Trajectory Judge)Agent Model=Gemma42026.06 | 78 | 82 | 81 | 86 | 88 | 68 | 74 | 74 | 80 | 83 | 162.54 | |
| GIFAgent Model=GPT-B2026.06 | 77 | 88 | 90 | 92 | 92 | 62 | 77 | 80 | 83 | 82 | 6.75 | |
| GIFAgent Model=Qwen-A2026.06 | 77 | 85 | 88 | 89 | 90 | 73 | 79 | 84 | 86 | 87 | 23.72 | |
| GIFAgent Model=Qwen-B2026.06 | 74 | 81 | 82 | 89 | 91 | 67 | 74 | 77 | 84 | 86 | 8.89 | |
| GPT (Full Trajectory Judge)Agent Model=Average2026.06 | 73 | 74 | 76 | 82 | 87 | 62 | 64 | 67 | 76 | 81 | 147.9 | |
| GPT (Full Trajectory Judge)Agent Model=Qwen-A2026.06 | 70 | 70 | 72 | 75 | 86 | 62 | 63 | 68 | 74 | 84 | 157.38 | |
| GPT (Full Trajectory Judge)Agent Model=Qwen-B2026.06 | 67 | 68 | 72 | 75 | 81 | 54 | 56 | 61 | 67 | 73 | 161.71 | |
| GPT (Full Trajectory Judge)Agent Model=Qwen-C2026.06 | 63 | 63 | 63 | 68 | 76 | 45 | 45 | 48 | 56 | 67 | 141.47 |