Specification Synthesis on VERINA
86.6Pass@1VERISPECGEN
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| VERISPECGENModel=Claude Opus 4.5, Model Category=Closed-Source Models2026.04 | 86.6 | 27.6 | 46.8 | |
| VERISPECGENModel=Claude Sonnet 4.5, Model Category=Closed-Source Models2026.04 | 72 | 21.3 | 42 | |
| VERISPECGENModel=Qwen3-Coder-480B-A35B, Model Category=Open-Weight Models2026.04 | 68.5 | 31.8 | 86.7 | |
| BaselineModel=Claude Opus 4.5, Model Category=Closed-Source Models2026.04 | 59 | — | — | |
| BaselineModel=Claude Sonnet 4.5, Model Category=Closed-Source Models2026.04 | 50.7 | — | — | |
| VERISPECGENModel=Qwen3-Coder-30B-A3B, Model Category=Open-Weight Models2026.04 | 50.3 | 21 | 71.7 | |
| BaselineModel=Qwen3-Coder-480B-A35B, Model Category=Open-Weight Models2026.04 | 36.7 | — | — | |
| BaselineModel=Qwen3-Coder-30B-A3B, Model Category=Open-Weight Models2026.04 | 29.3 | — | — | |
| VERISPECGENModel=Qwen3-4B-Instruct-2507, Model Category=Open-Weight Models2026.04 | 23 | 7.2 | 45.6 | |
| BaselineModel=Qwen3-4B-Instruct-2507, Model Category=Open-Weight Models2026.04 | 15.8 | — | — |