Verdict on FormalRx (test)
83.2AccuracyFormalRx-8B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| FormalRx-8B2026.07 | 83.2 | 93.2 | 83.6 | 88.1 | |
| Claude-Sonnet-4.6Model Type=Frontier Models2026.07 | 82.8 | 91.7 | 84.5 | 88 | |
| GPT-5-miniModel Type=Frontier Models2026.07 | 80.6 | 83.2 | 92.6 | 87.7 | |
| Claude-Sonnet-4Model Type=Frontier Models2026.07 | 80.1 | 85.2 | 88.7 | 86.9 | |
| GPT-5.3-CodexModel Type=Frontier Models2026.07 | 79.1 | 89.7 | 81.2 | 85.3 | |
| GPT-5.2Model Type=Frontier Models2026.07 | 77.9 | 82.1 | 89.9 | 85.8 | |
| DeepSeek-v3.2Model Type=Frontier Models2026.07 | 75.4 | 75.2 | 99.8 | 85.8 | |
| Qwen3-4BModel Type=Instruct Models2026.07 | 74.8 | 75.2 | 98.9 | 85.4 | |
| GPT-4.1Model Type=Frontier Models2026.07 | 74.1 | 89.9 | 74.2 | 81 | |
| Qwen3-8BModel Type=Instruct Models2026.07 | 73.9 | 79.7 | 87.3 | 83.3 | |
| Qwen3-1.7BModel Type=Instruct Models2026.07 | 73.5 | 75.5 | 95.3 | 84.3 | |
| LeanScorerModel Type=Specialized Alignment Metrics2026.07 | 66.6 | 64 | 68.2 | 63.2 | |
| BLEU (best F1)Model Type=Specialized Alignment Metrics2026.07 | 26.9 | 25.5 | 97 | 40.4 |