Safety Judge Performance Evaluation on HAJailBench 1.0 (test)
0.7589Kappa ScoreGPT-4o
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-4oModel=GPT-4o, Algorithm=Baseline Judge2025.11 | 0.7589 | 9.03 | — | |
| Multi-Agent JudgeModel=Qwen3-14B, Algorithm=Multi-Agent Judge2025.11 | 0.7331 | 4.13 | 46 | |
| Multi-Agent JudgeModel=Qwen3-8B, Algorithm=Multi-Agent Judge2025.11 | 0.6532 | 4.05 | 45 | |
| Multi-Agent JudgeModel=Qwen3-4B, Algorithm=Multi-Agent Judge2025.11 | 0.644 | 2.67 | 30 | |
| JailJudgeModel=Qwen3-8B, Algorithm=JailJudge2025.11 | 0.5626 | 4.58 | 51 | |
| JailJudgeModel=Qwen3-14B, Algorithm=JailJudge2025.11 | 0.5545 | 4.3 | 48 | |
| Multi-Agent JudgeModel=Qwen2.5-7B, Algorithm=Multi-Agent Judge2025.11 | 0.5233 | 1.62 | 18 | |
| JailJudgeModel=Qwen3-4B, Algorithm=JailJudge2025.11 | 0.5121 | 2.71 | 30 | |
| JailJudgeModel=Qwen2.5-7B, Algorithm=JailJudge2025.11 | 0.4334 | 2.51 | 28 | |
| ShieldGemma-9BModel=ShieldGemma-9B, Algorithm=Fine-tuned Judge2025.11 | 0.351 | — | — | |
| ShieldGemma-2BModel=ShieldGemma-2B, Algorithm=Fine-tuned Judge2025.11 | 0.0521 | — | — | |
| LlamaGuard3-8BModel=LlamaGuard3-8B, Algorithm=Fine-tuned Judge2025.11 | -0.0056 | 0.12 | 1 |