Hardware Vulnerability Detection on RTL Hardware Vulnerability
100AccuracyGemini 2.5 Flash
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini 2.5 FlashFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 100 | 42.86 | |
| Gemini 2.5 ProFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 100 | 50 | |
| Avg. for the SoTA LLMsFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 90.48 | 40.48 | |
| Gemini 2.0 FlashFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 85.71 | 35.71 | |
| Gemini 2.5 Flash LiteFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 85.71 | 50 | |
| GPT-4oFramework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 85.71 | 35.71 | |
| GPT-5Framework=SecureRAG-RTL, Model Scale=SoTA LLMs2026.03 | 85.71 | 28.57 | |
| Qwen 3 8BFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 71.43 | 35.71 | |
| Gemma 3 4B ItFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 64.29 | 42.86 | |
| LLaMA 3.2 11B V-ItFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 64.29 | 28.57 | |
| Gemini 2.5 FlashFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 57.14 | — | |
| GPT-5Framework=Baseline, Model Scale=SoTA LLMs2026.03 | 57.14 | — | |
| Avg. for medium LLMsFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 52.38 | 22.62 | |
| Qwen 2.5 Coder 3B ItFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 50 | 28.57 | |
| Code Llama 7B ItFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 50 | 21.43 | |
| Gemini 2.0 FlashFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 50 | — | |
| Gemini 2.5 ProFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 50 | — | |
| GPT-4oFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 50 | — | |
| Avg. for the SoTA LLMsFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 50 | — | |
| Falcon 3 3B ItFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 42.86 | 28.57 | |
| Granite 4.0 Micro 3BFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 42.86 | 28.57 | |
| StarCoder 2 7BFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 42.86 | 14.29 | |
| DeepSeek-Coder 7B ItFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 42.86 | 14.29 | |
| Mistral v0.3 7BFramework=SecureRAG-RTL, Model Scale=medium LLMs2026.03 | 42.86 | 21.43 | |
| Avg. for small LLMsFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 40.48 | 26.19 | |
| Qwen 3 8BFramework=Baseline, Model Scale=medium LLMs2026.03 | 35.71 | — | |
| LLaMA 3.2 11B V-ItFramework=Baseline, Model Scale=medium LLMs2026.03 | 35.71 | — | |
| Gemini 2.5 Flash LiteFramework=Baseline, Model Scale=SoTA LLMs2026.03 | 35.71 | — | |
| Avg. for medium LLMsFramework=Baseline, Model Scale=medium LLMs2026.03 | 29.76 | — | |
| Code Llama 7B ItFramework=Baseline, Model Scale=medium LLMs2026.03 | 28.57 | — | |
| StarCoder 2 7BFramework=Baseline, Model Scale=medium LLMs2026.03 | 28.57 | — | |
| DeepSeek-Coder 7B ItFramework=Baseline, Model Scale=medium LLMs2026.03 | 28.57 | — | |
| DeepSeek-R1-D-Qwen 1.5BFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 21.43 | 14.29 | |
| Qwen 2.5 Coder 3B ItFramework=Baseline, Model Scale=small LLMs2026.03 | 21.43 | — | |
| Phi 3 Mini 3.8B ItFramework=SecureRAG-RTL, Model Scale=small LLMs2026.03 | 21.43 | 14.29 | |
| Gemma 3 4B ItFramework=Baseline, Model Scale=small LLMs2026.03 | 21.43 | — | |
| Mistral v0.3 7BFramework=Baseline, Model Scale=medium LLMs2026.03 | 21.43 | — | |
| Falcon 3 3B ItFramework=Baseline, Model Scale=small LLMs2026.03 | 14.29 | — | |
| Granite 4.0 Micro 3BFramework=Baseline, Model Scale=small LLMs2026.03 | 14.29 | — | |
| Avg. for small LLMsFramework=Baseline, Model Scale=small LLMs2026.03 | 14.29 | — | |
| DeepSeek-R1-D-Qwen 1.5BFramework=Baseline, Model Scale=small LLMs2026.03 | 7.14 | — | |
| Phi 3 Mini 3.8B ItFramework=Baseline, Model Scale=small LLMs2026.03 | 7.14 | — |