Jailbreak Robustness on StrongReject (Detailed Attack Metrics)
67Direct Attack RateNo Defense
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No DefenseModel Family=Zephyr-7B2025.10 | 67 | 86 | 55 | 60 | 80 | 67 | 80 | — | 56 | 43 | 76 | 67 | — | |
| No DefenseModel Family=Mistral-7B2025.10 | 28 | 79 | 44 | 53 | 39 | 60 | 73 | — | 59 | 17 | 63 | 51 | — | |
| CATModel Family=Mistral-7B2025.10 | 20 | 57 | 36 | 32 | 29 | 0 | 67 | — | 65 | 8 | 63 | 38 | — | |
| WildJailbreakModel Family=Mistral-7B2025.10 | 6 | 41 | 16 | 22 | 20 | 15 | 11 | — | 27 | 7 | 55 | 22 | — | |
| WildJailbreakModel Family=Zephyr-7B2025.10 | 3 | 41 | 19 | 5 | 7 | 15 | 27 | — | 44 | 25 | 46 | 23 | — | |
| No DefenseModel Family=Llama3.1-8B2025.10 | 2 | 71 | 31 | 9 | 7 | 57 | 43 | — | 66 | 15 | 77 | 38 | — | |
| WildJailbreakModel Family=Llama3.1-8B2025.10 | 2 | 35 | 15 | 14 | 7 | 17 | 19 | — | 25 | 16 | 37 | 19 | — | |
| LATModel Family=Mistral-7B2025.10 | 2 | 46 | 19 | 6 | 3 | 0 | 41 | — | 2 | 2 | 68 | 19 | — | |
| Refusal TrainingModel Family=Llama3.1-8B2025.10 | 0 | 30 | 8 | 3 | 0 | 17 | 28 | — | 22 | 4 | 34 | 15 | — | |
| PAP TrainedModel Family=Llama3.1-8B2025.10 | 0 | 26 | 0 | 5 | 3 | 29 | 23 | — | 23 | 7 | 1 | 12 | — | |
| CATModel Family=Llama3.1-8B2025.10 | 0 | 50 | 16 | 4 | 5 | 1 | 37 | — | 27 | 5 | 62 | 20 | — | |
| CAT*Model Family=Llama3.1-8B2025.10 | 0 | 21 | 7 | 0 | 0 | 0 | 41 | — | 14 | 4 | 43 | 13 | — | |
| LATModel Family=Llama3.1-8B2025.10 | 0 | 39 | 7 | 0 | 0 | 0 | 38 | — | 49 | 0 | 37 | 17 | — | |
| LAT*Model Family=Llama3.1-8B2025.10 | 0 | 24 | 2 | 0 | 0 | 0 | 20 | — | 11 | 0 | 13 | 7 | — | |
| ASCoT (closed)Model Family=Llama3.1-8B2025.10 | 0 | 11 | 2 | 0 | 0 | 10 | 7 | — | 9 | 6 | 8 | 5 | — | |
| ASCoT (open)Model Family=Llama3.1-8B2025.10 | 0 | 11 | 3 | 0 | 1 | 12 | 10 | — | 12 | 12 | 8 | 7 | — | |
| Refusal TrainingModel Family=Zephyr-7B2025.10 | 0 | 60 | 19 | 7 | 2 | 24 | 57 | — | 35 | 9 | 43 | 26 | — | |
| PAP TrainedModel Family=Zephyr-7B2025.10 | 0 | 58 | 0 | 12 | 12 | 24 | 65 | — | 32 | 24 | 1 | 23 | — | |
| CATModel Family=Zephyr-7B2025.10 | 0 | 14 | 13 | 0 | 0 | 0 | 3 | — | 6 | 0 | 71 | 11 | — | |
| CAT*Model Family=Zephyr-7B2025.10 | 0 | 35 | 10 | 0 | 0 | 5 | 13 | — | 27 | 9 | 0 | 10 | — | |
| LATModel Family=Zephyr-7B2025.10 | 0 | 34 | 17 | 0 | 0 | 0 | 16 | — | 1 | 12 | 42 | 12 | — | |
| LAT*Model Family=Zephyr-7B2025.10 | 0 | 41 | 23 | 3 | 26 | 0 | 0 | — | 0 | 6 | 36 | 14 | — | |
| ASCoT (closed)Model Family=Zephyr-7B2025.10 | 0 | 10 | 1 | 0 | 5 | 12 | 36 | — | 5 | 12 | 9 | 9 | — | |
| ASCoT (open)Model Family=Zephyr-7B2025.10 | 0 | 10 | 2 | 0 | 0 | 11 | 25 | — | 40 | 18 | 9 | 12 | — | |
| Refusal TrainingModel Family=Mistral-7B2025.10 | 0 | 60 | 20 | 20 | 20 | 24 | 54 | — | 42 | 11 | 39 | 30 | — | |
| PAP TrainedModel Family=Mistral-7B2025.10 | 0 | 56 | 0 | 27 | 30 | 23 | 63 | — | 50 | 8 | 23 | 28 | — | |
| CAT*Model Family=Mistral-7B2025.10 | 0 | 33 | 16 | 3 | 0 | 5 | 31 | — | 31 | 6 | 57 | 18 | — | |
| LAT*Model Family=Mistral-7B2025.10 | 0 | 53 | 20 | 1 | 1 | 0 | 50 | — | 30 | 30 | 71 | 26 | — | |
| ASCoT (closed)Model Family=Mistral-7B2025.10 | 0 | 15 | 5 | 9 | 18 | 13 | 28 | — | 9 | 3 | 7 | 11 | — | |
| ASCoT (open)Model Family=Mistral-7B2025.10 | 0 | 20 | 4 | 26 | 27 | 13 | 26 | — | 2 | 6 | 11 | 14 | — | |
| CBBRModel=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 54.1 | |
| CBBRModel=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 64.4 | |
| CBBRModel=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 71.5 | |
| CBBRModel=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 49.2 | |
| DPRModel=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 35.1 | |
| DPRModel=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 43.5 | |
| DPRModel=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 41.2 | |
| DPRModel=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 57.8 | |
| No Defense (--)Model=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 57.2 | |
| No Defense (--)Model=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 70.9 | |
| No Defense (--)Model=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 76.2 | |
| No Defense (--)Model=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 72 | |
| OBBRModel=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 25.9 | |
| OBBRModel=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 30.7 | |
| OBBRModel=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 34.5 | |
| OBBRModel=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 33.4 | |
| ParaphraseModel=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 57.5 | |
| ParaphraseModel=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 58 | |
| ParaphraseModel=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 58.8 | |
| ParaphraseModel=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 73.3 | |
| Pre-PIAModel=Llama-3.2-1B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 2.7 | |
| Pre-PIAModel=Qwen2.5-1.5B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 27.2 | |
| Pre-PIAModel=Qwen2.5-7B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 20.3 | |
| Pre-PIAModel=Llama-3.1-8B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 2.1 |