Math Word Problem Solving on GSM8K (test set)
93.7AccuracyCAPO
Evaluation Results
| Method | Links | |
|---|---|---|
| CAPOFramework=promptolution, LLM=Gemma-3-27B-it2025.12 | 93.7 | |
| EvoPromptFramework=promptolution, LLM=Gemma-3-27B-it2025.12 | 91 | |
| AutoDiffFramework=AdalFlow, LLM=Gemma-3-27B-it2025.12 | 88.7 | |
| Full-FTBackbone=Qwen3-8B-Base, KV Sharing=Not Supported2026.02 | 85.8 | |
| PrefillShareBackbone=Qwen3-8B-Base, KV Sharing=Supported2026.02 | 84.8 | |
| GEPAFramework=DSPy, LLM=Gemma-3-27B-it2025.12 | 84.7 | |
| CoTCompilation=+ensemble, Training=trainset, Model=GPT-3.52023.10 | 81.6 | |
| unoptimizedFramework=Baseline, LLM=Gemma-3-27B-it2025.12 | 78.1 | |
| reflectionCompilation=bootstrap, Training=trainset, Model=GPT-3.52023.10 | 76 | |
| CoTCompilation=bootstrap, Training=trainset, Model=GPT-3.52023.10 | 72.9 | |
| CoTCompilation=fewshot, Training=+human_CoT, Model=GPT-3.52023.10 | 72.4 | |
| PrefillShareBackbone=LLaMA3.1-8B, KV Sharing=Supported2026.02 | 71.4 | |
| Full-FTBackbone=LLaMA3.1-8B, KV Sharing=Not Supported2026.02 | 71.3 | |
| OPROFramework=promptolution, LLM=Gemma-3-27B-it2025.12 | 69.7 | |
| vanillaCompilation=+ensemble, Training=trainset, Model=GPT-3.52023.10 | 61.9 | |
| vanillaCompilation=bootstrap×2, Training=trainset, Model=GPT-3.52023.10 | 61.7 | |
| PASERPruning Scheme=LLM-Pruner (25%)2025.02 | 49.4 | |
| PASERPruning Scheme=Wanda (2:4)2025.02 | 48.5 | |
| PASERPruning Scheme=SliceGPT (25%)2025.02 | 47.8 | |
| PASERPruning Scheme=SparseGPT (50%)2025.02 | 47.2 | |
| NuggetsPruning Scheme=LLM-Pruner (25%)2025.02 | 47.1 | |
| reflectionCompilation=+ensemble, Training=trainset, Model=Llama2-13b-chat2023.10 | 46.9 | |
| IFDPruning Scheme=LLM-Pruner (25%)2025.02 | 46.8 | |
| Full DataPruning Scheme=LLM-Pruner (25%)2025.02 | 46.5 | |
| Instruction MiningPruning Scheme=LLM-Pruner (25%)2025.02 | 46.2 | |
| NuggetsPruning Scheme=Wanda (2:4)2025.02 | 46.2 | |
| Full DataPruning Scheme=Wanda (2:4)2025.02 | 45.9 | |
| RandomPruning Scheme=LLM-Pruner (25%)2025.02 | 45.8 | |
| IFDPruning Scheme=Wanda (2:4)2025.02 | 45.8 | |
| NuggetsPruning Scheme=SparseGPT (50%)2025.02 | 45.7 | |
| NuggetsPruning Scheme=SliceGPT (25%)2025.02 | 45.4 | |
| Instruction MiningPruning Scheme=Wanda (2:4)2025.02 | 45.4 | |
| IFDPruning Scheme=SparseGPT (50%)2025.02 | 45.4 | |
| Full DataPruning Scheme=SparseGPT (50%)2025.02 | 45.2 | |
| IFDPruning Scheme=SliceGPT (25%)2025.02 | 45.1 | |
| Instruction MiningPruning Scheme=SparseGPT (50%)2025.02 | 44.9 | |
| Full DataPruning Scheme=SliceGPT (25%)2025.02 | 44.8 | |
| RandomPruning Scheme=Wanda (2:4)2025.02 | 44.7 | |
| Instruction MiningPruning Scheme=SliceGPT (25%)2025.02 | 44.5 | |
| w/o TrainingPruning Scheme=LLM-Pruner (25%)2025.02 | 44.3 | |
| RandomPruning Scheme=SparseGPT (50%)2025.02 | 44.3 | |
| RandomPruning Scheme=SliceGPT (25%)2025.02 | 43.9 | |
| w/o TrainingPruning Scheme=Wanda (2:4)2025.02 | 43.8 | |
| w/o TrainingPruning Scheme=SparseGPT (50%)2025.02 | 43.1 | |
| w/o TrainingPruning Scheme=SliceGPT (25%)2025.02 | 42.5 | |
| reflectionCompilation=bootstrap, Training=trainset, Model=Llama2-13b-chat2023.10 | 40.2 | |
| vanillaCompilation=bootstrap×2, Training=trainset, Model=Llama2-13b-chat2023.10 | 36.5 | |
| vanillaCompilation=+ensemble, Training=trainset, Model=Llama2-13b-chat2023.10 | 34.6 | |
| CoTCompilation=fewshot, Training=+human_CoT, Model=Llama2-13b-chat2023.10 | 33.7 | |
| LLaMA3.1-8BKV Sharing=Inherent2026.02 | 25.9 | |
| vanillaCompilation=none, Training=n/a, Model=GPT-3.52023.10 | 25.2 | |
| Qwen3-8B-BaseKV Sharing=Inherent2026.02 | 11.8 | |
| vanillaCompilation=none, Training=n/a, Model=Llama2-13b-chat2023.10 | 9.4 |