Response Harmfulness Detection on HarmBench Suite (HarmBench, SafeRLHF, BeaverTails, XSTest, WildGuard)
0.8333Macro Avg F1COLAGUARD
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| COLAGUARDModel Size=8B2026.05 | 0.8333 | — | — | — | 0.8155 | |
| GuardReasonerModel Size=8B2026.05 | 0.8313 | — | — | — | 0.8122 | |
| COLAGUARDModel Size=3B2026.05 | 0.8256 | — | — | — | 0.8022 | |
| GuardReasonerModel Size=3B2026.05 | 0.8249 | — | — | — | 0.808 | |
| WildGuardModel Size=7B2026.05 | 0.81 | — | — | — | 0.7795 | |
| GuardReasonerModel Size=1B2026.05 | 0.8078 | — | — | — | 0.7906 | |
| MD-JudgeModel Size=7B2026.05 | 0.8004 | — | — | — | 0.7867 | |
| GPT-4o+CoTModel Size=Unknown2026.05 | 0.7434 | — | — | — | 0.7445 | |
| BeaverDamModel Size=7B2026.05 | 0.7348 | — | — | — | 0.766 | |
| LLaMA Guard 2Model Size=8B2026.05 | 0.717 | — | — | — | 0.6699 | |
| LLaMA Guard 3Model Size=8B2026.05 | 0.7115 | — | — | — | 0.6497 | |
| o1-previewModel Size=Unknown2026.05 | 0.6954 | — | — | — | 0.6922 | |
| HarmBench MistralModel Size=7B2026.05 | 0.6934 | — | — | — | 0.667 | |
| HarmBench LLaMAModel Size=13B2026.05 | 0.6632 | — | — | — | 0.6549 | |
| GPT-4oModel Size=Unknown2026.05 | 0.6588 | — | — | — | 0.6941 | |
| Aegis Guard PermissiveModel Size=7B2026.05 | 0.6146 | — | — | — | 0.6355 | |
| LLaMA GuardModel Size=7B2026.05 | 0.6 | — | — | — | 0.5827 | |
| Aegis Guard DefensiveModel Size=7B2026.05 | 0.5962 | — | — | — | 0.6279 | |
| ShieldGemmaModel Size=9B2026.05 | 0.576 | — | — | — | 0.5567 | |
| QwQ-previewModel Size=32B2026.05 | 0.5464 | — | — | — | 0.5773 | |
| ShieldGemmaModel Size=2B2026.05 | 0.3379 | — | — | — | 0.2724 | |
| COLAGUARDModel Size=3B, Hardware=1×H100 (80GB) GPU2026.05 | — | 318.9 | 13 | 2.5041 | — | |
| COLAGUARDModel Size=8B, Hardware=1×H100 (80GB) GPU2026.05 | — | 342 | 12.9 | 2.3601 | — | |
| GuardReasonerModel Size=3B, Hardware=1×H100 (80GB) GPU2026.05 | — | 3,801.03 | 281.96 | 0.2122 | — | |
| GuardReasonerModel Size=8B, Hardware=1×H100 (80GB) GPU2026.05 | — | 4,407.8 | 289.4 | 0.1838 | — |