Financial Safety Evaluation on FinRED R4: Consumer Rights
94.25ASRGPTFuzzer (Black-box)
Evaluation results
| Method | Links | |
|---|---|---|
| GPTFuzzer (Black-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 94.25 | |
| GPTFuzzer (Black-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 93.68 | |
| GPTFuzzer (Black-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 90.8 | |
| GPTFuzzer (Black-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 87.93 | |
| GPTFuzzer (Black-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 86.2 | |
| GCG (White-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 72.41 | |
| GPTFuzzer (Black-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 67.24 | |
| GCG (White-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 66.67 | |
| GPTFuzzer (Black-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 50.2 | |
| Direct RequestTarget Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 49.43 | |
| GPTFuzzer (Black-box)Target Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 47.59 | |
| GPTFuzzer (Black-box)Target Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 46.44 | |
| GCG (White-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 43.1 | |
| GCG (White-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 40.23 | |
| AutoDAN-TurboTarget Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 38.51 | |
| AutoDAN-TurboTarget Model=gemma3, Model Category=General-Purpose sLM2026.06 | 37.81 | |
| AutoDAN (White-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 36.78 | |
| GCG (White-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 33.91 | |
| AutoDAN (White-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 33.91 | |
| Direct RequestTarget Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 31.03 | |
| TAP (Black-box)Target Model=FinMA, Model Category=Finance-Specific sLM2026.06 | 31.03 | |
| AutoDAN-TurboTarget Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 30.46 | |
| AutoDAN (White-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 29.89 | |
| TAP (Black-box)Target Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 29.89 | |
| AutoDAN-TurboTarget Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 27.59 | |
| Direct RequestTarget Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 27.01 | |
| Direct RequestTarget Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 26.44 | |
| AutoDAN (White-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 26.44 | |
| TAP (Black-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 26.44 | |
| GCG (White-box)Target Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 25.86 | |
| AutoDAN (White-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 24.71 | |
| TAP (Black-box)Target Model=Qwen2.5, Model Category=General-Purpose sLM2026.06 | 24.71 | |
| AutoDAN (White-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 23.56 | |
| Direct RequestTarget Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 20.69 | |
| AutoDAN-TurboTarget Model=EXAONE, Model Category=General-Purpose sLM2026.06 | 20.11 | |
| AutoDAN-TurboTarget Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 18.39 | |
| Direct RequestTarget Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 17.82 | |
| Direct RequestTarget Model=qqWen-7B, Model Category=Finance-Specific sLM2026.06 | 16.09 | |
| Direct RequestTarget Model=gemma3, Model Category=General-Purpose sLM2026.06 | 12.64 | |
| TAP (Black-box)Target Model=qqWen-32B, Model Category=Finance-Specific sLM2026.06 | 12.07 | |
| AutoDAN-TurboTarget Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 11.43 | |
| AutoDAN (White-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 10.34 | |
| TAP (Black-box)Target Model=gemma3, Model Category=General-Purpose sLM2026.06 | 9.77 | |
| TAP (Black-box)Target Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 9.77 | |
| AutoDAN-TurboTarget Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 9.2 | |
| AutoDAN-TurboTarget Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 9.2 | |
| GCG (White-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 8.62 | |
| GPTFuzzer (Black-box)Target Model=Claude-4, Model Category=API-Based LLM2026.06 | 6.9 | |
| TAP (Black-box)Target Model=Gemini-Pro, Model Category=API-Based LLM2026.06 | 5.75 | |
| Direct RequestTarget Model=GPT-5, Model Category=API-Based LLM2026.06 | 4.6 | |
| Direct RequestTarget Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 4.6 | |
| Direct RequestTarget Model=Claude-4, Model Category=API-Based LLM2026.06 | 4.6 | |
| AutoDAN-TurboTarget Model=GPT-5, Model Category=API-Based LLM2026.06 | 4.08 | |
| Direct RequestTarget Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 4.02 | |
| GPTFuzzer (Black-box)Target Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 4.02 | |
| AutoDAN-TurboTarget Model=GPT-5-mini, Model Category=API-Based LLM2026.06 | 3.59 | |
| TAP (Black-box)Target Model=Llama3.1, Model Category=General-Purpose sLM2026.06 | 3.45 | |
| TAP (Black-box)Target Model=GPT-5, Model Category=API-Based LLM2026.06 | 3.45 | |
| AutoDAN-TurboTarget Model=Claude-4, Model Category=API-Based LLM2026.06 | 2.04 | |
| TAP (Black-box)Target Model=Gemini-Flash, Model Category=API-Based LLM2026.06 | 1.15 | |
| TAP (Black-box)Target Model=Claude-4, Model Category=API-Based LLM2026.06 | 0.57 | |
| GPTFuzzer (Black-box)Target Model=GPT-5, Model Category=API-Based LLM2026.06 | 0 |