Financial Safety Evaluation on FinRED R5: Compliance Evasion
96.73Accuracy/Success Rate (ASR)GPTFuzzer (Black-box)
Evaluation results
| Method | Links | |
|---|---|---|
| GPTFuzzer (Black-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 96.73 | |
| GPTFuzzer (Black-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 93.47 | |
| GPTFuzzer (Black-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 85.31 | |
| GPTFuzzer (Black-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 81.63 | |
| GCG (White-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 74.69 | |
| GPTFuzzer (Black-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 73.01 | |
| GPTFuzzer (Black-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 65.71 | |
| GPTFuzzer (Black-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 62.85 | |
| GCG (White-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 58.78 | |
| GCG (White-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 57.2 | |
| AutoDAN-TurboTarget Model=gemma3, Model Category=General-Purpose sLM2026.06 | 53.26 | |
| GCG (White-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 50.61 | |
| Direct RequestTarget Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 45.31 | |
| Direct RequestTarget Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 44.49 | |
| GCG (White-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 42.45 | |
| Direct RequestTarget Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 37.14 | |
| TAP (Black-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 35.92 | |
| AutoDAN-TurboTarget Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 33.88 | |
| Direct RequestTarget Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 31.84 | |
| AutoDAN (White-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 31.84 | |
| AutoDAN (White-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 31.02 | |
| AutoDAN (White-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 31.02 | |
| GCG (White-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 30.61 | |
| AutoDAN (White-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 28.16 | |
| GPTFuzzer (Black-box)Target Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 27.76 | |
| Direct RequestTarget Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 27.35 | |
| AutoDAN (White-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 27.35 | |
| Direct RequestTarget Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 26.53 | |
| AutoDAN-TurboTarget Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 26.53 | |
| AutoDAN (White-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 25.31 | |
| Direct RequestTarget Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 24.9 | |
| TAP (Black-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 24.08 | |
| TAP (Black-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 23.67 | |
| AutoDAN-TurboTarget Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 23.67 | |
| TAP (Black-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 22.04 | |
| AutoDAN-TurboTarget Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 21.63 | |
| AutoDAN (White-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 19.59 | |
| AutoDAN-TurboTarget Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 19.59 | |
| AutoDAN-TurboTarget Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 18.52 | |
| TAP (Black-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 17.55 | |
| TAP (Black-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 16.33 | |
| Direct RequestTarget Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 15.1 | |
| Direct RequestTarget Model=gemma3, Model Category=General-Purpose sLM2026.06 | 14.29 | |
| GCG (White-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 13.47 | |
| TAP (Black-box)Target Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 11.84 | |
| AutoDAN-TurboTarget Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 11.84 | |
| AutoDAN-TurboTarget Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 11.43 | |
| Direct RequestTarget Model=GPT-5, Model Category=API-Based LLM2026.06 | 9.39 | |
| Direct RequestTarget Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 7.76 | |
| AutoDAN-TurboTarget Model=GPT-5, Model Category=API-Based LLM2026.06 | 6.93 | |
| TAP (Black-box)Target Model=GPT-5, Model Category=API-Based LLM2026.06 | 6.86 | |
| AutoDAN-TurboTarget Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 5.86 | |
| TAP (Black-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 5.71 | |
| Direct RequestTarget Model=Claude-4, Model Category=API-Based LLM2026.06 | 4.9 | |
| TAP (Black-box)Target Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 4.49 | |
| AutoDAN-TurboTarget Model=Claude-4, Model Category=API-Based LLM2026.06 | 4.29 | |
| GPTFuzzer (Black-box)Target Model=Claude-4, Model Category=API-Based LLM2026.06 | 4.08 | |
| GPTFuzzer (Black-box)Target Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 4.08 | |
| TAP (Black-box)Target Model=Claude-4, Model Category=API-Based LLM2026.06 | 2.86 | |
| TAP (Black-box)Target Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 2.86 | |
| GPTFuzzer (Black-box)Target Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 2.04 | |
| GPTFuzzer (Black-box)Target Model=GPT-5, Model Category=API-Based LLM2026.06 | 0 |