Jailbreaking on AdvBench (BERT Score & Harmful Scores)
4.84BERT ScoreJULI
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| JULIModel=Qwen2.5-1.5B-INST2025.05 | 4.84 | 4.76 | 3.73 | |
| JULIModel=Llama3-3B-INST2025.05 | 4.81 | 4.66 | 3.68 | |
| FLIPAPI Compatibility=true2025.05 | 4.76 | 2.06 | 0.5 | |
| JULITarget Model=Llama2-7B-CHAT, Defense=DEEPALIGN, Evaluation Setting=open-source setting2025.05 | 4.75 | 4.72 | 3.98 | |
| EDAPI Compatibility=false2025.05 | 4.43 | 4.28 | 3.36 | |
| GCGAPI Compatibility=false2025.05 | 4.42 | 1.93 | 0.48 | |
| EDModel=Llama3-8B-INST2025.05 | 4.39 | 4.1 | 3.02 | |
| JULIModel=Gemini-2.5-Pro2025.05 | 4.37 | 4.19 | 3.19 | |
| JULIModel=Llama3-8B-INST2025.05 | 4.33 | 4.57 | 3.44 | |
| LINTAPI Compatibility=false2025.05 | 4.05 | 1.95 | 0.77 | |
| EDModel=Llama3-3B-INST2025.05 | 4 | 4.22 | 2.99 | |
| NaiveAPI Compatibility=true2025.05 | 3.96 | 1.54 | 0.45 | |
| FLIPModel=Llama3-3B-INST2025.05 | 3.95 | 2.8 | 0.81 | |
| JULIModel=Llama2-7B-CHAT2025.05 | 3.94 | 4.22 | 3.5 | |
| JULIAPI Compatibility=false2025.05 | 3.7 | 3.08 | 2.35 | |
| EDModel=Llama2-7B-CHAT2025.05 | 3.69 | 2.96 | 1.84 | |
| LINTModel=Qwen2.5-1.5B-INST2025.05 | 3.65 | 4.21 | 3.13 | |
| NaiveModel=Qwen2.5-1.5B-INST2025.05 | 3.63 | 2.79 | 2.02 | |
| FLIPModel=Llama3-8B-INST2025.05 | 3.54 | 2.87 | 0.87 | |
| FLIPModel=Llama3-8B-INST2025.05 | 3.54 | 2.87 | 0.87 | |
| WTSModel=Qwen2.5-1.5B-INST2025.05 | 3.54 | 3.82 | 2.62 | |
| LINTModel=Llama2-7B-CHAT2025.05 | 3.42 | 3.7 | 2.22 | |
| FLIPModel=Gemini-2.5-Flash2025.05 | 3.41 | 3.13 | 1.33 | |
| FLIPModel=Qwen2.5-1.5B-INST2025.05 | 3.34 | 2.44 | 0.79 | |
| EDModel=Qwen2.5-1.5B-INST2025.05 | 3.24 | 1.26 | 0.6 | |
| WTSAPI Compatibility=false2025.05 | 3.14 | 1.37 | 0.42 | |
| GCGModel=Qwen2.5-1.5B-INST2025.05 | 3.13 | 2.77 | 2.05 | |
| BaseAPI Compatibility=true2025.05 | 3.07 | 1.4 | 0.41 | |
| BaseModel=Qwen2.5-1.5B-INST2025.05 | 2.98 | 3.04 | 2.14 | |
| JULI (API)API Compatibility=true2025.05 | 2.95 | 1.92 | 0.75 | |
| JULIModel=Llama3-8B-INST2025.05 | 2.91 | 3.12 | 2.21 | |
| LINTModel=Llama3-3B-INST2025.05 | 2.65 | 3.68 | 2.16 | |
| LINTModel=Llama3-8B-INST2025.05 | 2.63 | 3.77 | 2.25 | |
| JULIModel=Gemini-2.5-Flash2025.05 | 2.58 | 2.52 | 1.74 | |
| WTSModel=Llama3-8B-INST2025.05 | 2.46 | 2.38 | 1.26 | |
| NaiveModel=Llama2-7B-CHAT2025.05 | 2.41 | 2.51 | 1.74 | |
| FLIPModel=Gemini-2.5-Pro2025.05 | 2.39 | 2.52 | 1.38 | |
| GCGModel=Llama3-3B-INST2025.05 | 2.28 | 1.71 | 0.76 | |
| NaiveModel=Gemini-2.5-Flash2025.05 | 2.09 | 2.09 | 1.29 | |
| NaiveModel=Llama3-8B-INST2025.05 | 2.07 | 1.48 | 0.41 | |
| NaiveModel=Llama3-8B-INST2025.05 | 2.07 | 1.48 | 0.41 | |
| NaiveModel=Llama3-3B-INST2025.05 | 1.98 | 1.21 | 0.17 | |
| FLIPModel=Llama2-7B-CHAT2025.05 | 1.92 | 1.04 | 0.07 | |
| WTSModel=Llama2-7B-CHAT2025.05 | 1.87 | 1.64 | 0.56 | |
| GCGModel=Llama3-8B-INST2025.05 | 1.82 | 1.38 | 0.35 | |
| WTSModel=Llama3-3B-INST2025.05 | 1.81 | 1.56 | 0.44 | |
| NaiveModel=Gemini-2.5-Pro2025.05 | 1.7 | 1.91 | 1.21 | |
| BaseModel=Llama3-8B-INST2025.05 | 1.64 | 1.4 | 0.39 | |
| BaseModel=Llama3-8B-INST2025.05 | 1.64 | 1.4 | 0.39 | |
| GCGModel=Llama2-7B-CHAT2025.05 | 1.56 | 1.4 | 0.44 | |
| BaseModel=Llama3-3B-INST2025.05 | 1.32 | 1.21 | 0.21 | |
| NaiveTarget Model=Llama2-7B-CHAT, Defense=DEEPALIGN, Evaluation Setting=open-source setting2025.05 | 1.08 | 1.01 | 0 | |
| BaseModel=Llama2-7B-CHAT2025.05 | 0.79 | 1.04 | 0.04 | |
| BaseModel=Gemini-2.5-Flash2025.05 | 0.64 | 1 | 0.02 | |
| BaseModel=Gemini-2.5-Pro2025.05 | 0.61 | 1.02 | 0.06 |