Agent Safety Judgment on ROME Contextual Ambiguity 1.0
91.92F1 ScoreARISE (Std)
Evaluation Results
| Method | Links | |
|---|---|---|
| ARISE (Std)Model=Claude 3.7 Sonnet2026.05 | 91.92 | |
| ARISE (Std)Model=DeepSeek-R12026.05 | 87 | |
| ARISE (Std)Model=GPT-4o2026.05 | 85.26 | |
| Abl. UnsafeModel=Claude 3.7 Sonnet2026.05 | 84.16 | |
| Abl. FlippedModel=DeepSeek-R12026.05 | 83.13 | |
| ARISE (Std)Model=DeepSeek-V32026.05 | 82.13 | |
| Zero-shot (Base)Model=DeepSeek-R12026.05 | 80.22 | |
| Abl. FlippedModel=Claude 3.7 Sonnet2026.05 | 79.6 | |
| ARISE (Std)Model=Qwen3 (235B)2026.05 | 78.61 | |
| Abl. UnsafeModel=DeepSeek-R12026.05 | 77.73 | |
| Zero-shot (Base)Model=Claude 3.7 Sonnet2026.05 | 74.47 | |
| Abl. FlippedModel=DeepSeek-V32026.05 | 73.68 | |
| Abl. UnsafeModel=GPT-4o2026.05 | 73.03 | |
| Abl. UnsafeModel=DeepSeek-V32026.05 | 72.5 | |
| ARISE (Std)Model=Qwen3 (8B)2026.05 | 71.7 | |
| Abl. UnsafeModel=Qwen3 (8B)2026.05 | 68.13 | |
| Abl. UnsafeModel=Qwen3 (235B)2026.05 | 67.72 | |
| Abl. FlippedModel=GPT-4o2026.05 | 61.75 | |
| Abl. FlippedModel=Qwen3 (235B)2026.05 | 57.89 | |
| Zero-shot (Base)Model=Qwen3 (8B)2026.05 | 57.14 | |
| Zero-shot (Base)Model=Qwen3 (235B)2026.05 | 57.14 | |
| Abl. FlippedModel=Qwen3 (8B)2026.05 | 48.75 | |
| Zero-shot (Base)Model=DeepSeek-V32026.05 | 47.62 | |
| Zero-shot (Base)Model=GPT-4o2026.05 | 44.56 |