SIEM Query Generation on Curated YAML-based detection rules
0.161BLEUSynRAG
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SynRAGRun=Run 32025.12 | 0.161 | 0.6307 | |
| SynRAGRun=Average2025.12 | 0.1287 | 0.6039 | |
| SynRAGRun=Run 12025.12 | 0.1243 | 0.6076 | |
| SynRAGRun=Run 22025.12 | 0.1008 | 0.5736 | |
| LLaMARun=Run 32025.12 | 0.0818 | 0.4587 | |
| LLaMARun=Run 12025.12 | 0.0777 | 0.5048 | |
| LLaMARun=Average2025.12 | 0.0702 | 0.4839 | |
| GemmaRun=Run 22025.12 | 0.0658 | 0.3845 | |
| GemmaRun=Run 12025.12 | 0.0575 | 0.317 | |
| DeepSeekRun=Run 32025.12 | 0.0568 | 0.3923 | |
| GemmaRun=Average2025.12 | 0.056 | 0.3591 | |
| GPTRun=Run 32025.12 | 0.0539 | 0.4439 | |
| LLaMARun=Run 22025.12 | 0.0512 | 0.4881 | |
| GPTRun=Average2025.12 | 0.0467 | 0.4283 | |
| DeepSeekRun=Average2025.12 | 0.0461 | 0.3793 | |
| GemmaRun=Run 32025.12 | 0.0446 | 0.3759 | |
| GPTRun=Run 12025.12 | 0.0431 | 0.4184 | |
| GPTRun=Run 22025.12 | 0.0431 | 0.4226 | |
| DeepSeekRun=Run 22025.12 | 0.042 | 0.3892 | |
| DeepSeekRun=Run 12025.12 | 0.0396 | 0.3565 | |
| ClaudeRun=Run 32025.12 | 0.0365 | 0.3417 | |
| ClaudeRun=Average2025.12 | 0.0304 | 0.3063 | |
| ClaudeRun=Run 22025.12 | 0.0275 | 0.2927 | |
| ClaudeRun=Run 12025.12 | 0.0272 | 0.2844 |