Vulnerability Detection on Reveal (test)
48.3PrecisionDISCO (MLM+CLR++NT-MLM)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| DISCO (MLM+CLR++NT-MLM)pre-training_objective=MLM + CLR + NT-MLM, negatives=hard negative samples2021.10 | 48.3 | 44.6 | 46.4 | |
| GraphCodeBERTtype=pre-trained encoder-only2021.10 | 47.9 | 43.9 | 45.8 | |
| DISCO (MLM)pre-training_objective=MLM only2021.10 | 45.5 | 31 | 36.9 | |
| CodeBERTtype=pre-trained encoder-only2021.10 | 44.6 | 45.8 | 45.2 | |
| RoBERTatype=pre-trained encoder-only2021.10 | 44.5 | 39 | 41.6 | |
| Transformerlayers=12, initialization=from scratch2021.10 | 41.6 | 45.3 | 43.4 | |
| DISCO (MLM+CLR±)pre-training_objective=MLM + CLR, input=source code sequence only2021.10 | 39.4 | 50.5 | 44.2 | |
| DISCO (MLM+CLR+)pre-training_objective=MLM + CLR with hard negatives2021.10 | 38.6 | 47.7 | 42.6 | |
| Devignarchitecture=GNN2021.10 | 34.6 | 26.7 | 29.9 | |
| REVEALarchitecture=GNN + SMOTE + triplet loss2021.10 | 30.8 | 60.9 | 41.3 | |
| SySeVRtype=DL-based VD tool2021.10 | 24.5 | 40.1 | 30.3 | |
| ExtensionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Linevul2024.08 | 20.65 | 12.62 | 15.67 | |
| ExtensionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Linevul2024.08 | 18.01 | 12.81 | 14.97 | |
| VulDeePeckertype=DL-based VD tool2021.10 | 17.7 | 13.9 | 15.7 | |
| InjectionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Reveal2024.08 | 15.43 | 61.46 | 24.66 | |
| ExtensionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Devign2024.08 | 15.2 | 38.18 | 21.75 | |
| MutationAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Reveal2024.08 | 15.01 | 57.96 | 23.85 | |
| ExtensionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Devign2024.08 | 14.97 | 37.21 | 21.35 | |
| ExtensionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Reveal2024.08 | 14.82 | 52.71 | 23.13 | |
| ExtensionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Reveal2024.08 | 14.76 | 50 | 22.8 | |
| ROSDLVD Model=Reveal2024.08 | 14.35 | 54.52 | 22.72 | |
| InjectionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Reveal2024.08 | 14.23 | 64.48 | 23.32 | |
| InjectionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Devign2024.08 | 13.67 | 54.28 | 21.83 | |
| InjectionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Devign2024.08 | 13.44 | 53.26 | 21.46 | |
| ROSDLVD Model=Devign2024.08 | 13.28 | 39.93 | 19.93 | |
| NoAugDLVD Model=Reveal2024.08 | 12.9 | 65.14 | 21.54 | |
| VulGenDLVD Model=Reveal2024.08 | 12.46 | 64.6 | 20.9 | |
| MutationAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Reveal2024.08 | 12.23 | 68.46 | 20.75 | |
| VGXDLVD Model=Reveal2024.08 | 11.69 | 65.62 | 19.84 | |
| InjectionAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Linevul2024.08 | 11.57 | 26.43 | 16.1 | |
| VGXDLVD Model=Devign2024.08 | 11.02 | 59.35 | 18.58 | |
| MutationAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Devign2024.08 | 10.98 | 53.44 | 18.22 | |
| MutationAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Devign2024.08 | 10.96 | 61.64 | 18.6 | |
| NoAugDLVD Model=Devign2024.08 | 10.59 | 47.23 | 17.3 | |
| VulGenDLVD Model=Devign2024.08 | 10.45 | 57.9 | 17.7 | |
| InjectionAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Linevul2024.08 | 9.99 | 25.79 | 14.4 | |
| ROSDLVD Model=Linevul2024.08 | 9.88 | 22.71 | 13.77 | |
| VGXDLVD Model=Linevul2024.08 | 9.51 | 87.24 | 11.47 | |
| MutationAugmentation LLM=CodeQwen 1.5-7B-Chat, DLVD Model=Linevul2024.08 | 9.48 | 30.43 | 14.45 | |
| MutationAugmentation LLM=ChatGPT 3.5 Turbo, DLVD Model=Linevul2024.08 | 7.8 | 22.62 | 11.6 | |
| NoAugDLVD Model=Linevul2024.08 | 5.57 | 16.98 | 8.39 | |
| VulGenDLVD Model=Linevul2024.08 | 5.31 | 98.09 | 10.08 |