Verdict Prediction on ConsistencyCheck
85.8AccuracyDeepSeek-V3.2
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DeepSeek-V3.22026.07 | 85.8 | 65.2 | 69.8 | 67.4 | |
| GPT-5-mini2026.07 | 84 | 81.6 | 67.7 | 74 | |
| Claude-Sonnet-42026.07 | 83.3 | 78.9 | 68.8 | 73.5 | |
| Claude-Sonnet-4.62026.07 | 82.9 | 92.4 | 49 | 64 | |
| GPT-5.3-Codex2026.07 | 82.2 | 72.6 | 72.6 | 72.6 | |
| GPT-4.12026.07 | 77.4 | 72.5 | 52.9 | 61.2 | |
| LeanScorer2026.07 | 76.8 | 73.2 | 49.1 | 58.8 | |
| GPT-5.22026.07 | 71 | 55.1 | 87.2 | 67.5 | |
| FormalRx-8BSize=8B2026.07 | 70.2 | 54.8 | 65.4 | 59.6 | |
| FormalRx-4BSize=4B2026.07 | 67.2 | 50.9 | 68.9 | 58.5 | |
| Qwen3-8BSize=8B2026.07 | 66.8 | 50.6 | 64 | 56.5 | |
| FormalRx-1.7BSize=1.7B2026.07 | 65.3 | 49 | 73 | 58.6 | |
| Qwen3-4BSize=4B2026.07 | 58.3 | 43 | 73.4 | 54.2 | |
| Qwen3-1.7BSize=1.7B2026.07 | 53.6 | 39.3 | 69.9 | 50.3 | |
| BLEU (Best F1)2026.07 | 41.3 | 35.9 | 94.5 | 52 |