# IoT security vulnerability classification on Mutation Dataset 1.0 (Experiment B) Source: [IoT security vulnerability classification on Mutation Dataset 1.0 (Experiment B)]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:08:22.176Z Evaluation of IoT security vulnerability classification performance on the Mutation Dataset version 1.0, Experiment B, using metrics such as WAC, SAC, and Total Correct. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | IoT security vulnerability classification | | Dataset (including any stated split) | Mutation Dataset 1.0 (Experiment B) | | Metric | WAC | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 12 of 12 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [WAC](): Higher is better; unit: SCALAR. - [SAC](): Higher is better; unit: PERCENT. - [WTC](): Higher is better; unit: SCALAR. - [STC](): Higher is better; unit: SCALAR. - [WCC](): Higher is better; unit: SCALAR. - [SCC](): Higher is better; unit: SCALAR. - [Total Correct](): Higher is better; unit: SCALAR. ## Recorded results | Method | WAC | Result date | Report paper | | --- | --- | --- | --- | | Llama-70b | 35.55 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 34.55 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 31.89 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 30.9 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 16.94 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 15.95 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 14.95 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 12.62 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 11.96 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 10.3 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 6.31 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 5.65 | Not recorded | 695b456353fdd5c361742f15 | ### Llama-70b WAC: 35.55. Result ID: 6996f7605698665ec4441b70. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Zero-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 34.55. Result ID: 6996f7605698665ec4441b73. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Zero-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 31.89. Result ID: 6996f7605698665ec4441b74. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"One-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 30.9. Result ID: 6996f7605698665ec4441b75. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Two-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Llama-70b WAC: 16.94. Result ID: 6996f7605698665ec4441b72. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Two-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 15.95. Result ID: 6996f7605698665ec4441b6d. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Zero-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 14.95. Result ID: 6996f7605698665ec4441b6f. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Two-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Llama-70b WAC: 12.62. Result ID: 6996f7605698665ec4441b71. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"One-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 11.96. Result ID: 6996f7605698665ec4441b6e. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"One-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 10.3. Result ID: 6996f7605698665ec4441b76. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Zero-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 6.31. Result ID: 6996f7605698665ec4441b78. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"Two-Shot","protocol":"single response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 5.65. Result ID: 6996f7605698665ec4441b77. [Reporting paper]() Recorded evaluation configuration: {"number\_of\_shots":"One-Shot","protocol":"single response"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.