# Jailbreaking Generative AI: Multivector Phishing Threats and Transformer based Defenses

Source: [Jailbreaking Generative AI: Multivector Phishing Threats and Transformer based Defenses](<https://www.sota2.com/research/paper/68fa7a081e209131cea80a81>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-03T05:00:04.568Z

Authors: Rina Mishra, Gaurav Varshney

Published: 2025-07-16T12:32:46.000Z

## Abstract

The rise of Generative AI (GenAI) has reshaped the cybersecurity landscape by enabling new attack vectors and lowering the barrier for executing advanced social engineering campaigns. This study conducts an empirical analysis of jailbreaking vulnerabilities in ChatGPT-4o-Mini, showing that novices can bypass safeguards to generate complete multivector phishing attacks across email, web, SMS, and voice channels. Controlled experiments reveal that role-based jailbreaks produce fully operational attack paths capable of credential harvesting. User studies further demonstrate the disruptive potential of GenAI: novice participants exhibited a 240\\% increase in perceived phishing competence, a 400\\% improvement in task completion rates, and a 57\\% reduction in implementation time when assisted by GenAI compared to traditional internet resources. To address these risks, a transformer-based detection framework was developed, achieving an F1-score of 0.9864 (XLNET) for identifying malicious prompts. The work underscores the urgency of strengthening LLM guardrails and provides an annotated dataset to support future defenses.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Phishing and Malicious Prompt Classification on Proposed Phishing and Malicious Prompt Dataset 21000](<https://www.sota2.com/research/sota/phishing-and-malicious-prompt-classification-on-proposed-phishing-and-malicious-prompt-dataset-21000.md>): Phishing and Malicious Prompt Classification; Proposed Phishing and Malicious Prompt Dataset 21000; Accuracy: 96.

- [Jailbreak and Phishing Prompt Classification on 60 Malicious Queries](<https://www.sota2.com/research/sota/jailbreak-and-phishing-prompt-classification-on-60-malicious-queries.md>): Jailbreak and Phishing Prompt Classification; 60 Malicious Queries; Accuracy (Jailbreak/Phishing): not recorded.

- [Jailbreak Prompt Classification on AdvBench 520 prompts](<https://www.sota2.com/research/sota/jailbreak-prompt-classification-on-advbench-520-prompts.md>): Jailbreak Prompt Classification; AdvBench 520 prompts; Accuracy: not recorded.

- [Phishing Prompt Classification on Phishing Prompt Dataset](<https://www.sota2.com/research/sota/phishing-prompt-classification-on-phishing-prompt-dataset.md>): Phishing Prompt Classification; Phishing Prompt Dataset; Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.12185>)

[PDF](<https://arxiv.org/pdf/2507.12185.pdf>)