# Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

Source: [Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models](<https://www.sota2.com/research/paper/69c60eff0636e5bf4afc84db>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-28T05:00:04.914Z

Authors: Hieu Xuan Le, Benjamin Goh, Quy Anh Tang

Published: 2026-03-26T08:47:53.000Z

## Abstract

Prompt attacks, including jailbreaks and prompt injections, pose a critical security risk to Large Language Model (LLM) systems. In production, guardrails must mitigate these attacks under strict low-latency constraints, resulting in a deployment gap in which lightweight classifiers and rule-based systems struggle to generalize under distribution shift, while high-capacity LLM-based judges remain too slow or costly for live enforcement. In this work, we examine whether lightweight, general-purpose LLMs can reliably serve as security judges under real-world production constraints. Through careful prompt and output design, lightweight LLMs are guided through a structured reasoning process involving explicit intent decomposition, safety-signal verification, harm assessment, and self-reflection. We evaluate our method on a curated dataset combining benign queries from real-world chatbots with adversarial prompts generated via automated red teaming (ART), covering diverse and evolving patterns. Our results show that general-purpose LLMs, such as gemini-2.0-flash-lite-001, can serve as effective low-latency judges for live guardrails. This configuration is currently deployed in production as a centralized guardrail service for public service chatbots in Singapore. We additionally evaluate a Mixture-of-Models (MoM) setting to assess whether aggregating multiple LLM judges improves prompt-attack detection performance relative to single-model judges, with only modest gains observed.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Prompt Attack Detection on Curated prompt-attack dataset](<https://www.sota2.com/research/sota/prompt-attack-detection-on-curated-prompt-attack-dataset.md>): Prompt Attack Detection; Curated prompt-attack dataset; Latency: 1.44.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.25176>)

[PDF](<https://arxiv.org/pdf/2603.25176.pdf>)
