Cybersecurity Knowledge Evaluation on CyberMetric
93.3AccuracyLlama-3.3-70B-Ins-DAP
Evaluation results
| Method | Links | |
|---|---|---|
| Llama-3.3-70B-Ins-DAPSize [B]=70, Base Alignment=SFT + RLHF, Extra Tuning=DAP2025.06 | 93.3 | |
| Mixtral-8×7B-InstructSize [B]=45, Base Alignment=SFT + DPO, Extra Tuning=None2025.06 | 91.1 | |
| DSR1D-Qwen-14B-DAPSize [B]=14, Base Alignment=KD + SFT, Extra Tuning=DAP2025.06 | 90.4 | |
| Falcon-180B-ChatSize [B]=180, Base Alignment=SFT (Chat-style), Extra Tuning=None2025.06 | 87.1 | |
| Llama-Primus-Base (Cybersecurity specialized)Size [B]=8, Base Alignment=SFT + RLHF, Extra Tuning=Continuous Pretraining2025.06 | 86.6 | |
| Llama-3.1-8B-DAPSize [B]=8, Base Alignment=None (pretrain only), Extra Tuning=DAP2025.06 | 85.8 | |
| Foundation-Sec-8B (Cybersecurity specialized)Size [B]=8, Base Alignment=SFT + RLHF, Extra Tuning=Continuous Pretraining2025.06 | 85.1 | |
| Mistral-7B-Instruct-v0.2Size [B]=7, Base Alignment=SFT + DPO, Extra Tuning=None2025.06 | 76.4 | |
| Gemma-1.1-7B-itSize [B]=7, Base Alignment=SFT + (novel) RLHF, Extra Tuning=None2025.06 | 75.8 | |
| Llama-3-8B-InstructSize [B]=8, Base Alignment=SFT + RLHF, Extra Tuning=None2025.06 | 73.1 | |
| Llama-3-8B-BaseSize [B]=8, Base Alignment=None (pretrain only), Extra Tuning=PEFT + Continuous Pretraining2025.06 | 48.5 |