Mathematics on Mathematics Suite (GSM8K, MathQA)
73GSM8K AccuracyOlmo 3 7B-PC
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Olmo 3 7B-PCBackbone=Olmo 3 7B, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 73 | 75 | 37 | |
| Qwen2.5 7B-PCBackbone=Qwen2.5 7B, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 72 | 74 | 42 | |
| Granite 3.1 8B-PCBackbone=Granite 3.1 8B, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 66 | 67 | 34 | |
| Llama 3.1 8B-PCBackbone=Llama 3.1 8B, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 61 | 63 | 29 | |
| Granite 8B Code-PCBackbone=Granite 8B Code, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 59 | 61 | 25 | |
| Olmo 3 7B-NLBackbone=Olmo 3 7B, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 59 | 61 | 34 | |
| Qwen2.5 7B-NLBackbone=Qwen2.5 7B, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 52 | 54 | 36 | |
| Mistral 7B v0.3-PCBackbone=Mistral 7B v0.3, Strategy=Pseudo-code tuned, Zero-shot=true, Greedy decoding=true2025.05 | 44 | 45 | 25 | |
| Llama 3.1 8B-NLBackbone=Llama 3.1 8B, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 33 | 33 | 21 | |
| Qwen2.5 7B-CPBackbone=Qwen2.5 7B, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 30 | 32 | 33 | |
| Olmo 3 7B-CPBackbone=Olmo 3 7B, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 25 | 25 | 23 | |
| Granite 3.1 8B-CPBackbone=Granite 3.1 8B, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 16 | 17 | 18 | |
| Llama 3.1 8B-CPBackbone=Llama 3.1 8B, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 14 | 14 | 18 | |
| Mistral 7B v0.3-NLBackbone=Mistral 7B v0.3, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 12 | 13 | 21 | |
| Mistral 7B v0.3-CPBackbone=Mistral 7B v0.3, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 10 | 11 | 14 | |
| Granite 8B Code-CPBackbone=Granite 8B Code, Strategy=Code-Prompted, Zero-shot=true, Greedy decoding=true2025.05 | 10 | 11 | 17 | |
| Granite 8B Code-NLBackbone=Granite 8B Code, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 9 | 9 | 28 | |
| Granite 3.1 8B-NLBackbone=Granite 3.1 8B, Strategy=NL-tuned, Zero-shot=true, Greedy decoding=true2025.05 | 2 | 2 | 19 |