Autoformalization on AdvancedMath
100Compilation Pass Rate@10Kimina-7B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Kimina-7BAugmentation=+CRAMF2025.08 | 100 | 1.2 | |
| Godel-V2-8BAugmentation=+CRAMF2025.08 | 100 | 1.2 | |
| Claude 4.1Augmentation=+CRAMF2025.08 | 99.4 | 0.6 | |
| Claude 4.1Augmentation=Base2025.08 | 98.8 | — | |
| Kimina-7BAugmentation=Base2025.08 | 98.8 | — | |
| Godel-V2-8BAugmentation=Base2025.08 | 98.8 | — | |
| Herald-7BAugmentation=+CRAMF2025.08 | 90.2 | 12.8 | |
| Gemini 2.5 FlashAugmentation=+CRAMF2025.08 | 82.1 | 3.5 | |
| Herald-7BAugmentation=Base2025.08 | 80 | — | |
| Gemini 2.5 FlashAugmentation=Base2025.08 | 79.3 | — | |
| Godel-V2-8BAugmentation=CRAMF2025.08 | 68.2 | 18 | |
| Kimina-7BAugmentation=CRAMF2025.08 | 66.5 | 8.5 | |
| Claude 4.1Augmentation=CRAMF2025.08 | 65.9 | 15.2 | |
| Kimina-7BAugmentation=Base2025.08 | 61.3 | — | |
| GPT-4oAugmentation=+CRAMF2025.08 | 59.1 | 21.9 | |
| Godel-V2-8BAugmentation=Base2025.08 | 57.8 | — | |
| Claude 4.1Augmentation=Base2025.08 | 57.2 | — | |
| Herald-7BAugmentation=CRAMF2025.08 | 51.4 | 30.8 | |
| Gemini 2.5 FlashAugmentation=CRAMF2025.08 | 50.9 | 15.9 | |
| GPT-4oAugmentation=Base2025.08 | 48.5 | — | |
| Deepseek-V3Augmentation=+CRAMF2025.08 | 48.2 | 52.1 | |
| Gemini 2.5 FlashAugmentation=Base2025.08 | 43.9 | — | |
| Herald-7BAugmentation=Base2025.08 | 39.3 | — | |
| GPT-4oAugmentation=CRAMF2025.08 | 34.7 | 52.2 | |
| Deepseek-V3Augmentation=CRAMF2025.08 | 32.1 | 62.1 | |
| Deepseek-V3Augmentation=Base2025.08 | 31.7 | — | |
| GPT-4oAugmentation=Base2025.08 | 22.8 | — | |
| Deepseek-V3Augmentation=Base2025.08 | 19.8 | — |