RTL generation on RTLLM 50 cases (test)
0.42Pass@1Claude-4.5-Sonnet
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Claude-4.5-Sonnettype=Commercial Model2025.11 | 0.42 | 0.44 | 0.44 | |
| VeriDispatcher (Qwen Emb.-w)Embedding Type=Qwen, Commercial Model Inclusion=Included, Classifier=MLP2025.11 | 0.38 | 0.48 | 0.5 | |
| VeriDispatcher (OpenAI Emb.-w)Embedding Type=OpenAI, Commercial Model Inclusion=Included, Classifier=MLP2025.11 | 0.38 | 0.46 | 0.5 | |
| VeriDispatcher (OpenAI Emb.-wo)Embedding Type=OpenAI, Commercial Model Inclusion=Excluded, Classifier=MLP2025.11 | 0.36 | 0.46 | 0.5 | |
| VeriDispatcher (Average+OpenAI Emb.-w)Embedding Type=Average+OpenAI, Commercial Model Inclusion=Included, Classifier=MLP2025.11 | 0.36 | 0.46 | 0.5 | |
| OriGen2025.11 | 0.34 | 0.42 | 0.48 | |
| CodeV-R1-Distill-Qwen-7B2025.11 | 0.32 | 0.46 | 0.46 | |
| VeriDispatcher (Qwen Emb.-wo)Embedding Type=Qwen, Commercial Model Inclusion=Excluded, Classifier=MLP2025.11 | 0.32 | 0.44 | 0.48 | |
| RTLCoder-DeepSeek2025.11 | 0.28 | 0.36 | 0.4 | |
| HaVen2025.11 | 0.26 | 0.42 | 0.42 | |
| VeriDispatcher (Average Emb.-wo)Embedding Type=Average, Commercial Model Inclusion=Excluded, Classifier=MLP2025.11 | 0.26 | 0.44 | 0.5 | |
| RTLCoder-Mistral2025.11 | 0.24 | 0.4 | 0.46 | |
| GPT-5type=Commercial Model2025.11 | 0.24 | 0.34 | 0.38 | |
| CodeV-DS-6.7B2025.11 | 0.22 | 0.38 | 0.44 | |
| CodeV-CL-7B2025.11 | 0.18 | 0.34 | 0.44 | |
| CodeV-QW-7B2025.11 | 0.04 | 0.28 | 0.4 |