Over-refusal on XSTest
0Overrefusal RateInitial
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| InitialModel Size=1.5B2026.01 | 0 | — | — | — | — | — | — | |
| SafePathModel Size=7B2026.01 | 0 | — | — | — | — | — | — | |
| UnalignedModel Family=LLaMA-3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Standard DPOModel Family=LLaMA-3-8B2026.05 | 0 | — | — | — | — | — | — | |
| SequentialModel Family=LLaMA-3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Sqrt-CompetenceModel Family=LLaMA-3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Curri-DPOModel Family=LLaMA-3-8B2026.05 | 0 | — | — | — | — | — | — | |
| UnalignedModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Standard DPOModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| SequentialModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Sqrt-CompetenceModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Curri-DPOModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| Staged-CompetenceModel Family=Qwen3-8B2026.05 | 0 | — | — | — | — | — | — | |
| UnalignedModel Family=Yi-1.5-9B2026.05 | 0 | — | — | — | — | — | — | |
| GPT-4oNBF steering=false2025.02 | 0.4 | — | — | — | — | — | — | |
| Staged-CompetenceModel Family=LLaMA-3-8B2026.05 | 0.4 | — | — | — | — | — | — | |
| Standard DPOModel Family=Yi-1.5-9B2026.05 | 1.2 | — | — | — | — | — | — | |
| SequentialModel Family=Yi-1.5-9B2026.05 | 1.2 | — | — | — | — | — | — | |
| Curri-DPOModel Family=Yi-1.5-9B2026.05 | 1.2 | — | — | — | — | — | — | |
| Sqrt-CompetenceModel Family=Yi-1.5-9B2026.05 | 2.4 | — | — | — | — | — | — | |
| GPT-4oNBF steering=true2025.02 | 2.6 | — | — | — | — | — | — | |
| No DefenseModel Family=Zephyr-7B2025.10 | 3 | — | — | — | — | — | — | |
| Staged-CompetenceModel Family=Yi-1.5-9B2026.05 | 3.6 | — | — | — | — | — | — | |
| o1NBF steering=false2025.02 | 3.9 | — | — | — | — | — | — | |
| sonnet-4-thinking2025.10 | 4 | — | — | — | — | — | — | |
| VanillaBackbone=LLaVA-1.5-7B2025.10 | 4 | — | — | — | — | — | — | |
| ECSOBackbone=LLaVA-1.5-7B2025.10 | 4.8 | — | — | — | — | — | — | |
| WildJailbreakModel Family=Zephyr-7B2025.10 | 5 | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Zephyr-7B2025.10 | 5 | — | — | — | — | — | — | |
| No DefenseModel Family=Mistral-7B2025.10 | 5 | — | — | — | — | — | — | |
| WildJailbreakModel Family=Mistral-7B2025.10 | 5 | — | — | — | — | — | — | |
| Claude 3.5 SonnetNBF steering=false2025.02 | 5.2 | — | — | — | — | — | — | |
| ASTRABackbone=LLaVA-1.5-7B2025.10 | 5.4 | — | — | — | — | — | — | |
| o1NBF steering=true2025.02 | 5.7 | — | — | — | — | — | — | |
| o4-mini2025.10 | 6 | — | — | — | — | — | — | |
| ASCoTBase Model=LLaMA-3.1-8B, Access=open2025.10 | 6 | — | — | — | — | — | — | |
| No DefenseModel Family=Llama3.1-8B2025.10 | 6 | — | — | — | — | — | — | |
| WildJailbreakModel Family=Llama3.1-8B2025.10 | 6 | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Llama3.1-8B2025.10 | 6 | — | — | — | — | — | — | |
| PAP TrainedModel Family=Zephyr-7B2025.10 | 6 | — | — | — | — | — | — | |
| Improved CoTModel Series=7B Models, Safety Training Data=10002025.05 | 6.7 | — | — | — | — | — | — | |
| CATModel Family=Mistral-7B2025.10 | 7 | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Mistral-7B2025.10 | 7 | — | — | — | — | — | — | |
| ASCoT (open)Model Family=Mistral-7B2025.10 | 7 | — | — | — | — | — | — | |
| MoRASBackbone=LLaVA-1.5-7B2025.10 | 7.2 | — | — | — | — | — | — | |
| Improved CoTModel Series=32B Models, Safety Training Data=10002025.05 | 7.3 | — | — | — | — | — | — | |
| Claude 3.5 SonnetNBF steering=true2025.02 | 7.4 | — | — | — | — | — | — | |
| GPT-3.5-turboNBF steering=false2025.02 | 7.8 | — | — | — | — | — | — | |
| GPT-3.5-turboNBF steering=true2025.02 | 7.8 | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Zephyr-7B2025.10 | 9 | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Mistral-7B2025.10 | 9 | — | — | — | — | — | — | |
| Refusal TrainingModel Family=Llama3.1-8B2025.10 | 10 | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Zephyr-7B2025.10 | 10 | — | — | — | — | — | — | |
| PAP TrainedModel Family=Mistral-7B2025.10 | 10 | — | — | — | — | — | — | |
| PAP TrainedModel Family=Llama3.1-8B2025.10 | 12 | — | — | — | — | — | — | |
| ASCoT (closed)Model Family=Llama3.1-8B2025.10 | 12 | — | — | — | — | — | — | |
| STAR1-R1-Distill-32BModel Series=32B Models, Safety Training Data=10002025.05 | 12.4 | — | — | — | — | — | — | |
| ETABackbone=LLaVA-1.5-7B2025.10 | 15.6 | — | — | — | — | — | — | |
| RealSafe-R1-32BModel Series=32B Models, Safety Training Data=150002025.05 | 16 | — | — | — | — | — | — | |
| CoCABackbone=LLaVA-1.5-7B2025.10 | 24.8 | — | — | — | — | — | — | |
| LATModel Family=Mistral-7B2025.10 | 30 | — | — | — | — | — | — | |
| LAT*Model Family=Mistral-7B2025.10 | 30 | — | — | — | — | — | — | |
| CATModel Family=Llama3.1-8B2025.10 | 32 | — | — | — | — | — | — | |
| LAT*Model Family=Zephyr-7B2025.10 | 34 | — | — | — | — | — | — | |
| STAR1-R1-Distill-7BModel Series=7B Models, Safety Training Data=10002025.05 | 34 | — | — | — | — | — | — | |
| RealSafe-R1-7BModel Series=7B Models, Safety Training Data=150002025.05 | 36 | — | — | — | — | — | — | |
| SafeChainModel Size=1.5B2026.01 | 40 | — | — | — | — | — | — | |
| SafePathModel Size=1.5B2026.01 | 40 | — | — | — | — | — | — | |
| SafeChainModel Size=7B2026.01 | 40 | — | — | — | — | — | — | |
| THINKSAFEModel Size=7B2026.01 | 40 | — | — | — | — | — | — | |
| InitialModel Size=8B2026.01 | 40 | — | — | — | — | — | — | |
| SafePathModel Size=8B2026.01 | 40 | — | — | — | — | — | — | |
| CAT*Model Family=Mistral-7B2025.10 | 40 | — | — | — | — | — | — | |
| LATModel Family=Zephyr-7B2025.10 | 44 | — | — | — | — | — | — | |
| FigStepBackbone=LLaVA-1.5-7B2025.10 | 46 | — | — | — | — | — | — | |
| CAT*Model Family=Zephyr-7B2025.10 | 47 | — | — | — | — | — | — | |
| CAT*Model Family=Llama3.1-8B2025.10 | 49 | — | — | — | — | — | — | |
| AutoSteerBackbone=LLaVA-1.5-7B2025.10 | 62.4 | — | — | — | — | — | — | |
| DPO_WhispererAlignment=DPO, Data Recipe=ear-whisperer, Base Model=SFT_DB2025.05 | 80.67 | — | — | — | — | — | — | |
| LATModel Family=Llama3.1-8B2025.10 | 86 | — | — | — | — | — | — | |
| Mixtral (SFT_OG)LLM=Mixtral, SFT Status=SFT on original response data without CoTs2025.05 | 87.6 | — | — | — | — | — | — | |
| Mixtral (SFT_DB)LLM=Mixtral, SFT Status=SFT on AIDSAFE-generated CoTs and responses2025.05 | 91.84 | — | — | — | — | — | — | |
| SFT_DBTraining=Supervised Fine-Tuning, Data Generation=AIDSAFE2025.05 | 91.84 | — | — | — | — | — | — | |
| Qwen (SFT_DB)LLM=Qwen, SFT Status=SFT on AIDSAFE-generated CoTs and responses2025.05 | 93.6 | — | — | — | — | — | — | |
| Qwen (SFT_OG)LLM=Qwen, SFT Status=SFT on original response data without CoTs2025.05 | 98 | — | — | — | — | — | — | |
| LAT*Model Family=Llama3.1-8B2025.10 | 98 | — | — | — | — | — | — | |
| CATModel Family=Zephyr-7B2025.10 | 98 | — | — | — | — | — | — | |
| Mixtral (Base)LLM=Mixtral, SFT Status=None2025.05 | 98.8 | — | — | — | — | — | — | |
| Qwen (Base)LLM=Qwen, SFT Status=None2025.05 | 99.2 | — | — | — | — | — | — | |
| STAR-1Model Size=1.5B2026.01 | 120 | — | — | — | — | — | — | |
| THINKSAFEModel Size=1.5B2026.01 | 120 | — | — | — | — | — | — | |
| InitialModel Size=7B2026.01 | 120 | — | — | — | — | — | — | |
| SafeChainModel Size=8B2026.01 | 160 | — | — | — | — | — | — | |
| THINKSAFEModel Size=8B2026.01 | 160 | — | — | — | — | — | — | |
| STAR-1Model Size=7B2026.01 | 240 | — | — | — | — | — | — | |
| SafeKeyModel Size=1.5B2026.01 | 360 | — | — | — | — | — | — | |
| SafeKeyModel Size=7B2026.01 | 720 | — | — | — | — | — | — | |
| SafeKeyModel Size=8B2026.01 | 800 | — | — | — | — | — | — | |
| DirectRefusalModel Size=1.5B2026.01 | 840 | — | — | — | — | — | — | |
| STAR-1Model Size=8B2026.01 | 1,200 | — | — | — | — | — | — |