RTL code generation on RTLLM v2.0 (test)
76.2Syntax Pass@1o1-preview
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| o1-preview2025.05 | 76.2 | 83.71 | 84 | 50 | 60.86 | 62.52 | |
| GPT-4o2025.05 | 72 | 77.31 | 78.53 | 49.7 | 56.8 | 58.84 | |
| DeepRTL2Backbone=DeepSeek2025.05 | 71.6 | 80.58 | 81.75 | 38.5 | 52.62 | 55.99 | |
| DeepRTL2Backbone=Llama2025.05 | 68.3 | 81.31 | 83.36 | 33.7 | 49.57 | 52.9 | |
| DeepRTL2^1stBackbone=Llama2025.05 | 67.9 | 77.53 | 79.52 | 43.7 | 49.98 | 50 | |
| DeepRTLModel Size=16b2025.05 | 63.79 | 74.82 | 80.05 | 38.91 | 47.24 | 51.72 | |
| DeepRTL2^1stBackbone=DeepSeek2025.05 | 63.5 | 76.74 | 80.1 | 39.7 | 51.96 | 54.7 | |
| DeepRTLModel Size=220m2025.05 | 60.69 | 78.81 | 80.88 | 28.79 | 45.86 | 49.66 | |
| DeepRTL2^1st-DirectBackbone=DeepSeek2025.05 | 60.6 | 73.12 | 75.7 | 32.5 | 44.42 | 47.96 | |
| DeepSeek-Coder2025.05 | 59.3 | 72.38 | 74.67 | 31.4 | 39.59 | 42.57 | |
| CodeVBackbone=DeepSeek2025.05 | 57.6 | 80.23 | 83.25 | 30 | 49.63 | 54.74 | |
| GPT-3.52025.05 | 56.5 | 69.72 | 71.75 | 30.1 | 39.59 | 41.4 | |
| DeepRTL2^1st-DirectBackbone=Llama2025.05 | 54.48 | 63.52 | 67.99 | 16.28 | 28.78 | 32.76 | |
| CodeVBackbone=CodeQwen2025.05 | 51.5 | 77.71 | 82.17 | 23.1 | 44.54 | 52.22 | |
| CodeVBackbone=CodeLlama2025.05 | 47.7 | 74.96 | 82.2 | 22 | 39.49 | 45.74 | |
| Llama-3.12025.05 | 32.4 | 57.01 | 62.76 | 14.6 | 26.04 | 30.16 |