Multi-label content safety classification on Beavertails
0.86F1 ScoreQwen3Guard
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3GuardSize=8B, Mode=strict, Reasoning=false2025.12 | 0.86 | — | |
| Qwen3GuardSize=8B, Mode=loose, Reasoning=false2025.12 | 0.85 | — | |
| IBM Granite Guardian 3.3Size=8B, Reasoning=false2025.12 | 0.84 | — | |
| AprielGuardSize=8B, Reasoning=false2025.12 | 0.84 | — | |
| AprielGuardSize=8B, Reasoning=true2025.12 | 0.83 | — | |
| IBM Granite Guardian 3.2Size=5B, Reasoning=false2025.12 | 0.82 | — | |
| IBM Granite Guardian 3.3Size=8B, Reasoning=true2025.12 | 0.82 | — | |
| gpt-oss-safeguardSize=20B, Reasoning=true2025.12 | 0.81 | — | |
| IBM Granite Guardian 3.1Size=2B, Reasoning=false2025.12 | 0.8 | — | |
| IBM Granite Guardian 3.2Size=3B, Reasoning=false2025.12 | 0.8 | — | |
| Llama Guard 2Reasoning=false2025.12 | 0.73 | — | |
| Llama Guard 4Reasoning=false2025.12 | 0.71 | — | |
| Llama Guard 3Reasoning=false2025.12 | 0.69 | — | |
| ShieldGemmaSize=9B, Reasoning=false2025.12 | 0.65 | — | |
| Marginal ProbabilityModel=LLaMA Guard 32025.11 | 0.449 | 0.805 | |
| Marginal ProbabilityModel=LLaMA Guard 22025.11 | 0.442 | 0.705 | |
| Probability UncertaintyModel=LLaMA Guard 22025.11 | 0.431 | — | |
| Entropy UncertaintyModel=LLaMA Guard 22025.11 | 0.431 | — | |
| Greedy GenerationModel=LLaMA Guard 22025.11 | 0.43 | — | |
| LogTokUModel=LLaMA Guard 22025.11 | 0.43 | — | |
| Conditional ProbabilityModel=LLaMA Guard 22025.11 | 0.43 | 0.649 | |
| Joint ProbabilityModel=LLaMA Guard 22025.11 | 0.43 | 0.649 | |
| LogTokUModel=LLaMA 3.1 8B Instruct2025.11 | 0.426 | — | |
| Marginal ProbabilityModel=LLaMA 3.1 8B Instruct2025.11 | 0.424 | 0.809 | |
| Probability UncertaintyModel=LLaMA 3.1 8B Instruct2025.11 | 0.42 | — | |
| Entropy UncertaintyModel=LLaMA 3.1 8B Instruct2025.11 | 0.42 | — | |
| Joint ProbabilityModel=LLaMA Guard 32025.11 | 0.419 | 0.64 | |
| Greedy GenerationModel=LLaMA Guard 32025.11 | 0.418 | — | |
| LogTokUModel=LLaMA Guard 32025.11 | 0.418 | — | |
| Conditional ProbabilityModel=LLaMA Guard 32025.11 | 0.418 | 0.64 | |
| Probability UncertaintyModel=LLaMA Guard 32025.11 | 0.417 | — | |
| Entropy UncertaintyModel=LLaMA Guard 32025.11 | 0.417 | — | |
| Joint ProbabilityModel=LLaMA 3.1 8B Instruct2025.11 | 0.401 | 0.678 | |
| Conditional ProbabilityModel=LLaMA 3.1 8B Instruct2025.11 | 0.376 | 0.677 | |
| Greedy GenerationModel=LLaMA 3.1 8B Instruct2025.11 | 0.373 | — |