# Autoformalization on MiniF2F Source: [Autoformalization on MiniF2F]() Collection index: [llms.txt]() Data updated: 2026-03-24T10:34:34.101Z Evaluation of model performance on the Autoformalization task using the MiniF2F dataset, measured by Compilation Pass Rate@10 and Relative Gain. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Autoformalization | | Dataset (including any stated split) | MiniF2F | | Metric | Compilation Pass Rate@10 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 28 of 28 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Compilation Pass Rate@10](): Higher is better; unit: PERCENT. - [Relative Gain](): Higher is better; unit: PERCENT. ## Recorded results | Method | Compilation Pass Rate@10 | Result date | Report paper | | --- | --- | --- | --- | | Godel-V2-8B | 100 | Not recorded | 68fa7a7f1e209131cea837fb | | Godel-V2-8B | 99.6 | Not recorded | 68fa7a7f1e209131cea837fb | | Kimina-7B | 99.2 | Not recorded | 68fa7a7f1e209131cea837fb | | Kimina-7B | 98.8 | Not recorded | 68fa7a7f1e209131cea837fb | | Claude 4.1 | 98 | Not recorded | 68fa7a7f1e209131cea837fb | | Claude 4.1 | 95.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Godel-V2-8B | 95.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Herald-7B | 93.6 | Not recorded | 68fa7a7f1e209131cea837fb | | Claude 4.1 | 92.6 | Not recorded | 68fa7a7f1e209131cea837fb | | Gemini 2.5 Flash | 89.3 | Not recorded | 68fa7a7f1e209131cea837fb | | Godel-V2-8B | 88.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Kimina-7B | 84.8 | Not recorded | 68fa7a7f1e209131cea837fb | | Claude 4.1 | 83.6 | Not recorded | 68fa7a7f1e209131cea837fb | | GPT-4o | 82.7 | Not recorded | 68fa7a7f1e209131cea837fb | | Gemini 2.5 Flash | 82.4 | Not recorded | 68fa7a7f1e209131cea837fb | | Gemini 2.5 Flash | 82 | Not recorded | 68fa7a7f1e209131cea837fb | | Kimina-7B | 80.3 | Not recorded | 68fa7a7f1e209131cea837fb | | Herald-7B | 79.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Gemini 2.5 Flash | 72.1 | Not recorded | 68fa7a7f1e209131cea837fb | | GPT-4o | 70.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Deepseek-V3 | 69.3 | Not recorded | 68fa7a7f1e209131cea837fb | | Herald-7B | 63.1 | Not recorded | 68fa7a7f1e209131cea837fb | | GPT-4o | 60.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Deepseek-V3 | 52.3 | Not recorded | 68fa7a7f1e209131cea837fb | | GPT-4o | 49.6 | Not recorded | 68fa7a7f1e209131cea837fb | | Herald-7B | 49.2 | Not recorded | 68fa7a7f1e209131cea837fb | | Deepseek-V3 | 47.1 | Not recorded | 68fa7a7f1e209131cea837fb | | Deepseek-V3 | 36.9 | Not recorded | 68fa7a7f1e209131cea837fb | ### Godel-V2-8B Compilation Pass Rate@10: 100. Result ID: 69c2681647f34ce924aa5b41. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Godel-V2-8B Compilation Pass Rate@10: 99.6. Result ID: 69c2681647f34ce924aa5b40. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Kimina-7B Compilation Pass Rate@10: 99.2. Result ID: 69c2681647f34ce924aa5b3f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Kimina-7B Compilation Pass Rate@10: 98.8. Result ID: 69c2681647f34ce924aa5b3e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Claude 4.1 Compilation Pass Rate@10: 98. Result ID: 69c2681647f34ce924aa5b3b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Claude 4.1 Compilation Pass Rate@10: 95.1. Result ID: 69c2681647f34ce924aa5b3a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Godel-V2-8B Compilation Pass Rate@10: 95.1. Result ID: 69c2681647f34ce924aa5b6b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Herald-7B Compilation Pass Rate@10: 93.6. Result ID: 69c2681647f34ce924aa5b3d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Claude 4.1 Compilation Pass Rate@10: 92.6. Result ID: 69c2681647f34ce924aa5b65. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Gemini 2.5 Flash Compilation Pass Rate@10: 89.3. Result ID: 69c2681647f34ce924aa5b39. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Godel-V2-8B Compilation Pass Rate@10: 88.1. Result ID: 69c2681647f34ce924aa5b6a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Kimina-7B Compilation Pass Rate@10: 84.8. Result ID: 69c2681647f34ce924aa5b69. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Claude 4.1 Compilation Pass Rate@10: 83.6. Result ID: 69c2681647f34ce924aa5b64. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### GPT-4o Compilation Pass Rate@10: 82.7. Result ID: 69c2681647f34ce924aa5b37. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Gemini 2.5 Flash Compilation Pass Rate@10: 82.4. Result ID: 69c2681647f34ce924aa5b38. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Gemini 2.5 Flash Compilation Pass Rate@10: 82. Result ID: 69c2681647f34ce924aa5b63. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Kimina-7B Compilation Pass Rate@10: 80.3. Result ID: 69c2681647f34ce924aa5b68. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Herald-7B Compilation Pass Rate@10: 79.1. Result ID: 69c2681647f34ce924aa5b3c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Gemini 2.5 Flash Compilation Pass Rate@10: 72.1. Result ID: 69c2681647f34ce924aa5b62. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### GPT-4o Compilation Pass Rate@10: 70.1. Result ID: 69c2681647f34ce924aa5b36. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Deepseek-V3 Compilation Pass Rate@10: 69.3. Result ID: 69c2681647f34ce924aa5b35. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"+CRAMF"}. This record does not assert independent reproduction. ### Herald-7B Compilation Pass Rate@10: 63.1. Result ID: 69c2681647f34ce924aa5b67. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### GPT-4o Compilation Pass Rate@10: 60.1. Result ID: 69c2681647f34ce924aa5b61. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Deepseek-V3 Compilation Pass Rate@10: 52.3. Result ID: 69c2681647f34ce924aa5b34. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### GPT-4o Compilation Pass Rate@10: 49.6. Result ID: 69c2681647f34ce924aa5b60. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Herald-7B Compilation Pass Rate@10: 49.2. Result ID: 69c2681647f34ce924aa5b66. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. ### Deepseek-V3 Compilation Pass Rate@10: 47.1. Result ID: 69c2681647f34ce924aa5b5f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"CRAMF"}. This record does not assert independent reproduction. ### Deepseek-V3 Compilation Pass Rate@10: 36.9. Result ID: 69c2681647f34ce924aa5b5e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Augmentation":"Base"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.