Cybersecurity Knowledge and Reasoning on CTI, SecEval, Cyber Metric 2000, CISSP, and Adv. CTI Suite
69.71SecEval ScoreCyberPal-2.0-14B
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| CyberPal-2.0-14Bzero-shot=true, temperature=0, parameters=14B2025.10 | 69.71 | 75.51 | 86 | 89.95 | 90.4 | 89.58 | 70.77 | 70.95 | 92.93 | 81.76 | |
| Llama 4 Scout 17B 16E Instructzero-shot=true, temperature=0, parameters=17B2025.10 | 67.84 | 69.46 | 71.95 | 92.5 | 87.37 | 79.01 | 66.48 | 68.58 | 75.32 | 75.39 | |
| Mistral Small 3.2 24B Instruct 2506zero-shot=true, temperature=0, parameters=24B2025.10 | 67.47 | 68.82 | 68.05 | 91.6 | 87.37 | 74.37 | 65.9 | 67.19 | 76.22 | 74.11 | |
| CyberPal-2.0-8BZero-shot=true, Temperature=0, Model Size=8B2025.10 | 66.93 | 75.15 | 85.95 | 89.85 | 88.89 | 87.61 | 71.06 | 70.26 | 87.66 | 80.37 | |
| Mixtral 8x22B v0.1zero-shot=true, temperature=0, parameters=8x22B2025.10 | 65.65 | 62.81 | 66.7 | 87.75 | 82.32 | 71.41 | 59.89 | 61.66 | 77.51 | 70.63 | |
| Microsoft Phi 4zero-shot=true, temperature=02025.10 | 63.73 | 68.22 | 64 | 91 | 83.33 | 66.76 | 68.48 | 64.03 | 60.28 | 69.98 | |
| DeepSeek V3zero-shot=true, temperature=0, parameters=685B2025.10 | 63.68 | 73.35 | 72.45 | 93.65 | 91.41 | 78.73 | 69.63 | 68.97 | 65.17 | 75.23 | |
| Primus-mergedZero-shot=true, Temperature=02025.10 | 59.06 | 65.2 | 63.9 | 85.1 | 78.28 | 64.92 | 55.3 | 50.77 | 59.98 | 64.72 | |
| Foundation-Sec-8B-InstructZero-shot=true, Temperature=0, Model Size=8B2025.10 | 54.81 | 63.24 | 67.95 | 84.5 | 69.69 | 68.87 | 60.74 | 55.52 | 57.31 | 64.74 | |
| Primus-reasoningZero-shot=true, Temperature=02025.10 | 53.03 | — | — | 86.05 | 73.23 | 64.78 | 53.58 | 52.24 | 58.79 | 63.01 | |
| Lily-Cybersecurity-7B-v0.2Zero-shot=true, Temperature=0, Model Size=7B2025.10 | 37.14 | 55.31 | 42.9 | 80 | 68.18 | 58.45 | 52.43 | 39.71 | 46.34 | 53.38 | |
| DeepHat-v1-7BZero-shot=true, Temperature=0, Model Size=7B2025.10 | 33.21 | 61.24 | 68.1 | 84 | 76.76 | 63.23 | 60.74 | 56.12 | 52.05 | 61.72 |