Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Framework
About
WebShell attacks - where adversaries implant malicious scripts on web servers - remain a persistent threat. Prior machine-learning and deep-learning detectors typically depend on task-specific supervision and can be brittle under data scarcity, rapid concept drift, and out-of-distribution (OOD) deployment. Large language models (LLMs) have recently shown strong code understanding capabilities, but their reliability for WebShell detection remains unclear. We address this gap by (i) systematically evaluating seven LLMs (including GPT-4, LLaMA-3.1-70B, and Qwen-2.5 variants) against representative sequence- and graph-based baselines on 26.59K PHP scripts, and (ii) proposing Behavioral Function-Aware Detection (BFAD), a behavior-centric framework that adapts LLM inference to WebShell-specific execution patterns. BFAD anchors analysis on security-sensitive PHP functions via a Critical Function Filter, constructs compact LLM inputs with Context-Aware Code Extraction, and selects in-context demonstrations using Weighted Behavioral Function Profiling, which ranks examples by a behavior-weighted, function-level similarity. Empirically, we observe a consistent precision-recall asymmetry: larger LLMs often achieve high precision but miss attacks (lower recall), while smaller models exhibit the opposite tendency; moreover, off-the-shelf LLM prompting underperforms established detectors. BFAD substantially improves all evaluated LLMs, boosting F1 by 13.82% on average; notably, GPT-4, LLaMA-3.1-70B, and Qwen-2.5-Coder-14B exceed prior SOTA benchmarks, while Qwen-2.5-Coder-3B becomes competitive with traditional methods. Overall, our results clarify when LLMs succeed or fail on WebShell detection, provide a practical recipe, and highlight future directions for making LLM-based detection more reliable.