Organizational target classification on Hacker Forum Exploit Posts OOD 2024+ (test)
98.65AccuracyTRACE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| TRACEGroup=Ours2020.12 | 98.65 | 97 | 96.63 | 97.38 | |
| BERTGroup=LLM2020.12 | 97.78 | 89.29 | 90.49 | 91.91 | |
| ModernBERTGroup=LLM2020.12 | 97.73 | 88.11 | 89.93 | 86.54 | |
| CySecBERTGroup=LLM2020.12 | 97.51 | 94.45 | 98.35 | 92.32 | |
| RoBERTaGroup=LLM2020.12 | 96.49 | 84.36 | 87.84 | 82.77 | |
| BiLSTM-AttnGroup=DL + FastText2020.12 | 96.43 | 80.54 | 80.35 | 80.8 | |
| TransformerGroup=DL + FastText2020.12 | 96.33 | 77.18 | 79.84 | 75.74 | |
| GRUGroup=DL + FastText2020.12 | 94.92 | 81.18 | 88.06 | 77.34 | |
| CNN-LSTMGroup=DL + FastText2020.12 | 94.65 | 72.33 | 72.87 | 71.89 | |
| BiLSTMGroup=DL + FastText2020.12 | 93.25 | 81.84 | 87.16 | 79.85 | |
| SVM (Linear)Group=Class. ML2020.12 | 92.65 | 75.69 | 75.5 | 76.26 | |
| Logistic Reg.Group=Class. ML2020.12 | 91.36 | 73.42 | 74.16 | 73.19 | |
| LSTMGroup=DL + FastText2020.12 | 90.87 | 61.45 | 65.78 | 59.73 | |
| Decision TreeGroup=Class. ML2020.12 | 88.55 | 62.43 | 70.26 | 58.58 | |
| BiLSTM-AttnGroup=DL + GloVe2020.12 | 82.6 | 71.65 | 71.2 | 74.51 | |
| GRUGroup=DL + GloVe2020.12 | 81.31 | 65.18 | 65.32 | 66.83 | |
| Naive BayesGroup=Class. ML2020.12 | 79.52 | 40.01 | 55.26 | 39.45 | |
| BiLSTMGroup=DL + GloVe2020.12 | 79.42 | 63.11 | 64.02 | 64.39 | |
| TransformerGroup=DL + GloVe2020.12 | 77.42 | 72.38 | 82.31 | 69.36 | |
| CNN-LSTMGroup=DL + GloVe2020.12 | 77.36 | 68.2 | 69.64 | 70.88 | |
| LSTMGroup=DL + GloVe2020.12 | 71.75 | 57.33 | 60.52 | 57.95 | |
| RNNGroup=DL + FastText2020.12 | 65.64 | 22.46 | 22.16 | 23.08 | |
| RNNGroup=DL + GloVe2020.12 | 22.42 | 10.42 | 11.87 | 18.54 |