Security Goal Extraction on SecGoal (test)
100RecallGemma2-9b-FT
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=NSSK2026.04 | 100 | 3 | 3 | 5 | 0 | 0 | 100 | 100 | |
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=Kao-Chow v12026.04 | 100 | 7 | 5 | 5 | 0 | 2 | 71.4 | 83.3 | |
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=SPDM2026.04 | 100 | 17 | 11 | 8 | 0 | 6 | 64.7 | 78.6 | |
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=OPC-UA2026.04 | 100 | 34 | 20 | 6 | 0 | 14 | 58.8 | 74.1 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=NSSK2026.04 | 100 | 5 | 4 | 5 | 0 | 1 | 80 | 88.9 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=Kao-Chow v12026.04 | 100 | 7 | 6 | 5 | 0 | 1 | 85.7 | 92.3 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=NSSK2026.04 | 100 | 26 | 6 | 5 | 0 | 20 | 23.1 | 37.5 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=Kao-Chow v12026.04 | 100 | 21 | 6 | 5 | 0 | 15 | 28.6 | 44.4 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=SPDM2026.04 | 100 | 375 | 20 | 8 | 0 | 355 | 5.3 | 10.1 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=OPC-UA2026.04 | 100 | 310 | 43 | 6 | 0 | 267 | 13.9 | 24.4 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=NSSK2026.04 | 100 | 28 | 6 | 5 | 0 | 22 | 21.4 | 35.3 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=Kao-Chow v12026.04 | 100 | 19 | 5 | 5 | 0 | 14 | 26.3 | 41.7 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=SPDM2026.04 | 100 | 397 | 21 | 8 | 0 | 377 | 5.3 | 10 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=OPC-UA2026.04 | 100 | 365 | 35 | 6 | 0 | 330 | 9.6 | 17.5 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=NSSK2026.04 | 100 | 24 | 6 | 5 | 0 | 18 | 25 | 40 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=Kao-Chow v12026.04 | 100 | 19 | 5 | 5 | 0 | 14 | 26.3 | 41.7 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=SPDM2026.04 | 100 | 430 | 27 | 8 | 0 | 403 | 6.3 | 11.8 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=OPC-UA2026.04 | 100 | 356 | 37 | 6 | 0 | 319 | 10.4 | 18.8 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=Average2026.04 | 97.1 | — | — | — | — | — | 16 | 26.7 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=Average2026.04 | 97.1 | — | — | — | — | — | 17.7 | 29 | |
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=Average2026.04 | 94.3 | — | — | — | — | — | 73.6 | 81.7 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=Average2026.04 | 94.3 | — | — | — | — | — | 18.9 | 30.3 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=Average2026.04 | 91.3 | — | — | — | — | — | 72.3 | 80.4 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=SPDM2026.04 | 87.5 | 15 | 8 | 7 | 1 | 7 | 53.3 | 66.3 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=EDHOC2026.04 | 85.7 | 30 | 22 | 12 | 2 | 8 | 73.3 | 79 | |
| Gemini2.5-FlashType=General Purpose (SOTA), Protocol=EDHOC2026.04 | 85.7 | 161 | 28 | 12 | 2 | 133 | 17.4 | 28.9 | |
| DeepSeek-V3.2ThinkingType=General Purpose (SOTA), Protocol=EDHOC2026.04 | 85.7 | 153 | 31 | 12 | 2 | 122 | 20.3 | 32.8 | |
| Qwen2.5-7b-FTType=Fine-Tuned (Ours), Protocol=OPC-UA2026.04 | 83.3 | 29 | 20 | 5 | 1 | 9 | 69 | 75.5 | |
| Gemma2-9b-FTType=Fine-Tuned (Ours), Protocol=EDHOC2026.04 | 71.4 | 31 | 22 | 10 | 4 | 9 | 71 | 71.2 | |
| DeepSeek-V3.2Type=General Purpose (SOTA), Protocol=EDHOC2026.04 | 71.4 | 137 | 32 | 10 | 4 | 105 | 23.4 | 35.2 |