Vulnerability Detection on CVE-Solidity (test)
59Top 1 Hit Rate (cs)OpenInterpreter
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| OpenInterpretermode=FT, threshold_t=0.52025.11 | 59 | 73 | — | — | |
| DeepSeekmode=FT, threshold_t=0.52025.11 | 56 | 82 | — | — | |
| NXCodemode=FT, threshold_t=0.52025.11 | 47 | 47 | — | — | |
| OpenInterpretermode=FT, threshold_t=0.72025.11 | 45 | 49 | — | — | |
| Code Llamamode=FT, threshold_t=0.52025.11 | 41 | 43 | — | — | |
| DeepSeekmode=Base, threshold_t=0.52025.11 | 38 | 77 | — | — | |
| Gemmamode=FT, threshold_t=0.52025.11 | 37 | 50 | — | — | |
| NXCodemode=FT, threshold_t=0.72025.11 | 34 | 34 | — | — | |
| Perm-Opt Ensemblemode=FT, threshold_t=0.52025.11 | 33 | 73 | — | — | |
| Code Llamamode=FT, threshold_t=0.72025.11 | 29 | 32 | — | — | |
| Gemmamode=FT, threshold_t=0.72025.11 | 29 | 37 | — | — | |
| Perm-Opt Ensemblemode=Base, threshold_t=0.52025.11 | 28 | 69 | — | — | |
| Weighted Ensemblemode=Base, threshold_t=0.52025.11 | 28 | 69 | — | — | |
| NXCodemode=Base, threshold_t=0.52025.11 | 26 | 68 | — | — | |
| Weighted Ensemblemode=FT, threshold_t=0.52025.11 | 25 | 84 | — | — | |
| OpenInterpretermode=FT, threshold_t=0.92025.11 | 24 | 25 | — | — | |
| DeepSeekmode=FT, threshold_t=0.72025.11 | 22 | 33 | — | — | |
| Gemmamode=FT, threshold_t=0.92025.11 | 21 | 21 | — | — | |
| Code Llamamode=Base, threshold_t=0.52025.11 | 20 | 44 | — | — | |
| OpenInterpretermode=Base, threshold_t=0.52025.11 | 20 | 50 | — | — | |
| Perm-Opt Ensemblemode=FT, threshold_t=0.72025.11 | 16 | 49 | — | — | |
| Gemmamode=Base, threshold_t=0.52025.11 | 15 | 37 | — | — | |
| DeepSeekmode=Base, threshold_t=0.72025.11 | 12 | 20 | — | — | |
| NXCodemode=Base, threshold_t=0.72025.11 | 9 | 31 | — | — | |
| Code Llamamode=FT, threshold_t=0.92025.11 | 8 | 9 | — | — | |
| Weighted Ensemblemode=FT, threshold_t=0.72025.11 | 7 | 34 | — | — | |
| Perm-Opt Ensemblemode=Base, threshold_t=0.72025.11 | 6 | 32 | — | — | |
| NXCodemode=FT, threshold_t=0.92025.11 | 6 | 6 | — | — | |
| Code Llamamode=Base, threshold_t=0.72025.11 | 5 | 6 | — | — | |
| Weighted Ensemblemode=Base, threshold_t=0.72025.11 | 5 | 31 | — | — | |
| Perm-Opt Ensemblemode=FT, threshold_t=0.92025.11 | 5 | 20 | — | — | |
| OpenInterpretermode=Base, threshold_t=0.72025.11 | 4 | 18 | — | — | |
| Gemmamode=Base, threshold_t=0.72025.11 | 2 | 5 | — | — | |
| Code Llamamode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| DeepSeekmode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| DeepSeekmode=FT, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| Gemmamode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| NXCodemode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| OpenInterpretermode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| Perm-Opt Ensemblemode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| Weighted Ensemblemode=Base, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| Weighted Ensemblemode=FT, threshold_t=0.92025.11 | 0 | 0 | — | — | |
| Code Llamamode=Base, match_type=direct2025.11 | — | — | 6 | 12 | |
| Code Llamamode=FT, match_type=direct2025.11 | — | — | 27 | 31 | |
| DeepSeekmode=Base, match_type=direct2025.11 | — | — | 5 | 14 | |
| DeepSeekmode=FT, match_type=direct2025.11 | — | — | 38 | 56 | |
| Gemmamode=Base, match_type=direct2025.11 | — | — | 4 | 11 | |
| Gemmamode=FT, match_type=direct2025.11 | — | — | 28 | 38 | |
| NXCodemode=Base, match_type=direct2025.11 | — | — | 9 | 41 | |
| NXCodemode=FT, match_type=direct2025.11 | — | — | 34 | 34 | |
| OpenInterpretermode=Base, match_type=direct2025.11 | — | — | 6 | 27 | |
| OpenInterpretermode=FT, match_type=direct2025.11 | — | — | 39 | 41 | |
| Perm-Opt Ensemblemode=Base, match_type=direct2025.11 | — | — | 7 | 44 | |
| Perm-Opt Ensemblemode=FT, match_type=direct2025.11 | — | — | 40 | 60 | |
| Weighted Ensemblemode=Base, match_type=direct2025.11 | — | — | 6 | 44 | |
| Weighted Ensemblemode=FT, match_type=direct2025.11 | — | — | 9.3 | 56 |