IoT security vulnerability classification on Mutation Dataset 1.0 (Experiment B)
35.55WACLlama-70b
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Llama-70bnumber_of_shots=Zero-Shot, protocol=single response2026.01 | 35.55 | 638 | 25.5 | 15.52 | 38.68 | 0 | 19.04 | |
| Gemini-2.5-Pronumber_of_shots=Zero-Shot, protocol=single response2026.01 | 34.55 | 3,989 | 80.54 | 74.14 | 37.72 | 48.41 | 48.42 | |
| Gemini-2.5-Pronumber_of_shots=One-Shot, protocol=single response2026.01 | 31.89 | 5,479 | 72.15 | 76.44 | 47.51 | 60.86 | 55.55 | |
| Gemini-2.5-Pronumber_of_shots=Two-Shot, protocol=single response2026.01 | 30.9 | 5,824 | 18.79 | 87.36 | 46.39 | 55.46 | 48.5 | |
| Llama-70bnumber_of_shots=Two-Shot, protocol=single response2026.01 | 16.94 | 3,777 | 48.66 | 23.56 | 26.48 | 0 | 21.8 | |
| Llama-8bnumber_of_shots=Zero-Shot, protocol=single response2026.01 | 15.95 | 80 | 27.18 | 37.36 | 23.27 | 1.38 | 14.11 | |
| Llama-8bnumber_of_shots=Two-Shot, protocol=single response2026.01 | 14.95 | 0 | 24.83 | 29.31 | 28.41 | 0.69 | 14.11 | |
| Llama-70bnumber_of_shots=One-Shot, protocol=single response2026.01 | 12.62 | 798 | 46.98 | 13.79 | 37.24 | 0.28 | 18.68 | |
| Llama-8bnumber_of_shots=One-Shot, protocol=single response2026.01 | 11.96 | 106 | 23.49 | 13.22 | 40.61 | 6.36 | 17.31 | |
| Deepseek-r1-7bnumber_of_shots=Zero-Shot, protocol=single response2026.01 | 10.3 | 452 | 0.67 | 44.25 | 6.1 | 2.35 | 7.29 | |
| Deepseek-r1-7bnumber_of_shots=Two-Shot, protocol=single response2026.01 | 6.31 | 266 | 0.67 | 58.05 | 2.57 | 10.1 | 8.86 | |
| Deepseek-r1-7bnumber_of_shots=One-Shot, protocol=single response2026.01 | 5.65 | 133 | 0.67 | 65.52 | 2.09 | 4.84 | 7.45 |