# Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools? Source: [Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?]() Collection index: [llms.txt]() Data updated: 2026-01-06T05:00:04.580Z Authors: Jason Quantrill, Noura Khajehnouri, Zihan Guo, Manar H. Alalfi Published: 2026-01-02T04:17:36.000Z ## Abstract Smart home IoT platforms such as openHAB rely on Trigger Action Condition (TAC) rules to automate device behavior, but the interplay among these rules can give rise to interaction threats, unintended or unsafe behaviors emerging from implicit dependencies, conflicting triggers, or overlapping conditions. Identifying these threats requires semantic understanding and structural reasoning that traditionally depend on symbolic, constraint-driven static analysis. This work presents the first comprehensive evaluation of Large Language Models (LLMs) across a multi-category interaction threat taxonomy, assessing their performance on both the original openHAB (oHC/IoTB) dataset and a structurally challenging Mutation dataset designed to test robustness under rule transformations. We benchmark Llama 3.1 8B, Llama 70B, GPT-4o, Gemini-2.5-Pro, and DeepSeek-R1 across zero-, one-, and two-shot settings, comparing their results against oHIT's manually validated ground truth. Our findings show that while LLMs exhibit promising semantic understanding, particularly on action- and condition-related threats, their accuracy degrades significantly for threats requiring cross-rule structural reasoning, especially under mutated rule forms. Model performance varies widely across threat categories and prompt settings, with no model providing consistent reliability. In contrast, the symbolic reasoning baseline maintains stable detection across both datasets, unaffected by rule rewrites or structural perturbations. These results underscore that LLMs alone are not yet dependable for safety critical interaction-threat detection in IoT environments. We discuss the implications for tool design and highlight the potential of hybrid architectures that combine symbolic analysis with LLM-based semantic interpretation to reduce false positives while maintaining structural rigor. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Rule-Inconsistency Task Classification on RIT (Rule-Inconsistency Tasks)](): Rule-Inconsistency Task Classification; RIT (Rule-Inconsistency Tasks); WAC: not recorded. - [Requirements Inconsistency Classification on RIT 3-category taxonomy v1.0 (test)](): Requirements Inconsistency Classification; RIT 3-category taxonomy v1.0 (test); AC: not recorded. - [RIT Classification on IoT Security RIT dataset Experiment D](): RIT Classification; IoT Security RIT dataset Experiment D; Accuracy (AC): not recorded. - [3-category taxonomy classification (multiple responses allowed) on Mutation Dataset (Experiment C)](): 3-category taxonomy classification (multiple responses allowed); Mutation Dataset (Experiment C); Agreement Coefficient (AC): not recorded. - [IoT security static analysis vulnerability classification on Mutation Dataset Experiment D](): IoT security static analysis vulnerability classification; Mutation Dataset Experiment D; Accuracy (AC): not recorded. - [IoT security vulnerability classification on Mutation Dataset 1.0 (Experiment B)](): IoT security vulnerability classification; Mutation Dataset 1.0 (Experiment B); WAC: not recorded. - [Reasoning in IoT (RIT) Classification on Mutation Dataset (Experiment A)](): Reasoning in IoT (RIT) Classification; Mutation Dataset (Experiment A); WAC: not recorded. ## Original sources and implementation [Original paper]() [PDF]()