Cybersecurity Incident Response on ACDC
73.49RARDEFENGRAPH (Full)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| DEFENGRAPH (Full)LLM=GPT-4o2026.06 | 73.49 | 72.46 | 29.24 | 61 | 171 | 50 | 6 | |
| DEFENGRAPH (SKG + GRP + CF)LLM=GPT-4o2026.06 | 61.45 | 52.17 | 24.49 | 51 | 147 | 36 | 8 | |
| DEFENGRAPH (Full)LLM=LLaMA 3.1 253B2026.06 | 61.45 | 56.52 | 36.11 | 51 | 108 | 39 | 5 | |
| DEFENGRAPH (Full)LLM=QWen 3 235B2026.06 | 59.04 | 46.38 | 25.2 | 49 | 127 | 32 | 4 | |
| DEFENGRAPH (Full)LLM=DeepSeek R1 671B2026.06 | 56.63 | 59.42 | 36.94 | 47 | 111 | 41 | 11 | |
| DEFENGRAPH (SKG + GRP + CF)LLM=QWen 3 235B2026.06 | 51.81 | 42.03 | 24.58 | 43 | 118 | 29 | 5 | |
| DEFENGRAPH (SKG + GRP + CF)LLM=LLaMA 3.1 253B2026.06 | 46.99 | 42.03 | 29.01 | 39 | 100 | 29 | 7 | |
| GraphRAG (SKG)LLM=GPT-4o2026.06 | 45.78 | 39.63 | 11.19 | 38 | 134 | 15 | 8 | |
| DEFENGRAPH (SKG + GRP + CF)LLM=DeepSeek R1 671B2026.06 | 45.78 | 43.48 | 28.57 | 38 | 105 | 30 | 13 | |
| GraphRAG (SKG)LLM=DeepSeek R1 671B2026.06 | 38.23 | 25.81 | 7.36 | 26 | 95 | 7 | 14 | |
| GraphRAG (SKG)LLM=QWen 3 235B2026.06 | 35.29 | 29.03 | 7.26 | 24 | 124 | 9 | 8 | |
| No KB (LLM Only)LLM=GPT-4o2026.06 | 33.73 | 14.49 | 5.81 | 28 | 172 | 10 | 5 | |
| Hybrid RAGLLM=GPT-4o2026.06 | 33.65 | 20.29 | 8.19 | 27 | 171 | 14 | 6 | |
| GraphRAG (SKG)LLM=LLaMA 3.1 253B2026.06 | 27.94 | 19.35 | 6.85 | 19 | 73 | 5 | 9 | |
| Hybrid RAGLLM=QWen 3 235B2026.06 | 24.1 | 11.59 | 6.72 | 20 | 119 | 8 | 6 | |
| Hybrid RAGLLM=LLaMA 3.1 253B2026.06 | 12.05 | 8.71 | 4.76 | 10 | 126 | 6 | 6 | |
| Hybrid RAGLLM=DeepSeek R1 671B2026.06 | 10.84 | 10.14 | 5.65 | 9 | 124 | 7 | 8 |