# Mathematical Reasoning on AIME 2024 (Avg@k and Pass@k Evaluation) Source: [Mathematical Reasoning on AIME 2024 (Avg@k and Pass@k Evaluation)]() Collection index: [llms.txt]() Data updated: 2026-05-11T10:11:09.490Z Evaluation of model performance on the American Invitational Mathematics Examination (AIME) 2024 dataset, measured using Average Recall (avg@k) and Pass@k metrics for k=1 and k=8. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Mathematical Reasoning | | Dataset (including any stated split) | AIME competition-level 2024 | | Metric | Avg@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Avg@1](): Higher is better; unit: PERCENT. - [Avg@8](): Higher is better; unit: PERCENT. - [Pass@1](): Higher is better; unit: PERCENT. - [Pass@8](): Higher is better; unit: PERCENT. ## Recorded results | Method | Avg@1 | Result date | Report paper | | --- | --- | --- | --- | | GPT-5 nano | 73.3 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-8B | 70 | Not recorded | 6a0162badbc836b87d82332a | | gpt-oss-20b | 70 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-4B | 60 | Not recorded | 6a0162badbc836b87d82332a | | EngGPT2-16B-A3B | 53.3 | Not recorded | 6a0162badbc836b87d82332a | | Ministral-3-8B | 23.3 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-12b-it | 16.7 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-4b-it | 13.3 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.2-3B-Instruct | 3.3 | Not recorded | 6a0162badbc836b87d82332a | | Moonlight-16B-A3B-Instruct | 3.3 | Not recorded | 6a0162badbc836b87d82332a | | FastwebMIIA-7B | 0 | Not recorded | 6a0162badbc836b87d82332a | | Minerva-7B-instruct-v1.0 | 0 | Not recorded | 6a0162badbc836b87d82332a | | Velvet-14B | 0 | Not recorded | 6a0162badbc836b87d82332a | | LLaMAntino-3-ANITA-8B | 0 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.1-8B-Instruct | 0 | Not recorded | 6a0162badbc836b87d82332a | | deepseek-moe-16b-chat | 0 | Not recorded | 6a0162badbc836b87d82332a | ### GPT-5 nano Avg@1: 73.3. Result ID: 6a01ab3c47f34ce924e41e61. [Reporting paper]() Recorded evaluation configuration: {"Model Version":"nano"}. This record does not assert independent reproduction. ### Qwen3-8B Avg@1: 70. Result ID: 6a01ab3c47f34ce924e41e5b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"8B"}. This record does not assert independent reproduction. ### gpt-oss-20b Avg@1: 70. Result ID: 6a01ab3c47f34ce924e41e63. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"20b"}. This record does not assert independent reproduction. ### Qwen3-4B Avg@1: 60. Result ID: 6a01ab3c47f34ce924e41e5a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"4B"}. This record does not assert independent reproduction. ### EngGPT2-16B-A3B Avg@1: 53.3. Result ID: 6a01ab3c47f34ce924e41e55. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"16B"}. This record does not assert independent reproduction. ### Ministral-3-8B Avg@1: 23.3. Result ID: 6a01ab3c47f34ce924e41e60. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"8B"}. This record does not assert independent reproduction. ### gemma-3-12b-it Avg@1: 16.7. Result ID: 6a01ab3c47f34ce924e41e5f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"12b"}. This record does not assert independent reproduction. ### gemma-3-4b-it Avg@1: 13.3. Result ID: 6a01ab3c47f34ce924e41e5e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"4b"}. This record does not assert independent reproduction. ### Llama-3.2-3B-Instruct Avg@1: 3.3. Result ID: 6a01ab3c47f34ce924e41e5c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Parameters":"3B"}. This record does not assert independent reproduction. ### Moonlight-16B-A3B-Instruct Avg@1: 3.3. Result ID: 6a01ab3c47f34ce924e41e64. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"16B"}. This record does not assert independent reproduction. ### FastwebMIIA-7B Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e56. [Reporting paper]() Recorded evaluation configuration: {"Parameters":"7B"}. This record does not assert independent reproduction. ### Minerva-7B-instruct-v1.0 Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e57. [Reporting paper]() Recorded evaluation configuration: {"Parameters":"7B"}. This record does not assert independent reproduction. ### Velvet-14B Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e58. [Reporting paper]() Recorded evaluation configuration: {"Parameters":"14B"}. This record does not assert independent reproduction. ### LLaMAntino-3-ANITA-8B Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e59. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"8B"}. This record does not assert independent reproduction. ### Llama-3.1-8B-Instruct Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e5d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Parameters":"8B"}. This record does not assert independent reproduction. ### deepseek-moe-16b-chat Avg@1: 0. Result ID: 6a01ab3c47f34ce924e41e62. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Parameters":"16b"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.