Mathematical Reasoning on GSM8K (Accuracy, ASR)
93AccuracyAutoregressive
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 93 | — | — | |
| SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 93 | — | 83.5 | |
| AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 93 | — | — | |
| AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 93 | — | — | |
| SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 92 | — | 81.85 | |
| SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 92 | — | 82.75 | |
| DiversedTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 92 | — | 90.26 | |
| SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 92 | — | 84.15 | |
| SpecCascadeTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 92 | — | 84.43 | |
| DIVERSEDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 92 | — | 90.7 | |
| SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 92 | — | 84.15 | |
| SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 92 | — | 84.43 | |
| DIVERSEDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 92 | — | 90.7 | |
| Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 91 | — | 89.67 | |
| SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 91 | — | 59.58 | |
| Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 91 | — | 87.62 | |
| Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 91 | — | 87.62 | |
| AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 90 | — | — | |
| SD (Lossy)Target/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 90 | — | 85.02 | |
| SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 90 | — | 85.02 | |
| SCASModel=Qwen2.5-7B2026.05 | 89.38 | — | — | |
| DIVERSEDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 88 | — | 76.48 | |
| GRACEModel=Qwen2.5-7B2026.05 | 86.85 | — | — | |
| SD (lossy)Target/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 86 | — | 60.1 | |
| SpecCascadeTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 86 | — | 58.65 | |
| Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 86 | — | 67.58 | |
| HY-1.8B-FP16Model Scale=1.8B, Precision=FP16, Instruction-tuned=true2026.02 | 84.08 | — | — | |
| SCASModel=Qwen2.5-3B2026.05 | 82.82 | — | — | |
| Llama3.1Condition=Clean2025.01 | 82.25 | — | — | |
| Llama3.1Punctuation Percentage=10%2025.01 | 81.04 | — | — | |
| Llama3.1Punctuation Percentage=30%2025.01 | 78.84 | — | — | |
| HY-1.8B-INT4Model Scale=1.8B, Precision=INT4, Instruction-tuned=true2026.02 | 78.7 | — | — | |
| VoidPaddingBlock size B=5122026.06 | 78.62 | — | — | |
| VoidPaddingBlock size B=1282026.06 | 78.39 | — | — | |
| VoidPaddingBlock size B=Mean2026.06 | 78.27 | — | — | |
| VoidPaddingBlock size B=642026.06 | 77.79 | — | — | |
| MathstralCondition=Clean2025.01 | 77.63 | — | — | |
| HY-1.8B-2BitModel Scale=1.8B, Precision=2Bit, Instruction-tuned=true2026.02 | 77.33 | — | — | |
| VIGORBackbone=Qwen2.5-3B-Base, Training Dataset=CodeContests, RL Strategy=VIGOR2026.05 | 77.1 | — | — | |
| Llama3.1Punctuation Percentage=50%2025.01 | 77.02 | — | — | |
| GRACEModel=Qwen2.5-3B2026.05 | 76.8 | — | — | |
| OriginalBlock size B=642026.06 | 76.8 | — | — | |
| MathstralPunctuation Percentage=10%2025.01 | 75.51 | — | — | |
| Llama3Condition=Clean2025.01 | 75.43 | — | — | |
| INTUITORBackbone=Qwen2.5-3B-Base, Training Dataset=CodeContests, RL Strategy=INTUITOR2026.05 | 75.13 | — | — | |
| RainbowPaddingBlock size B=642026.06 | 74.53 | — | — | |
| DeepSeekCondition=Clean2025.01 | 73.76 | — | — | |
| DeepSeekPunctuation Percentage=10%2025.01 | 73.76 | — | — | |
| RainbowPaddingBlock size B=1282026.06 | 73.39 | — | — | |
| Llama3Punctuation Percentage=10%2025.01 | 73.31 | — | — | |
| Llama3Punctuation Percentage=30%2025.01 | 73.08 | — | — | |
| Llama3Punctuation Percentage=50%2025.01 | 72.17 | — | — | |
| RainbowPaddingBlock size B=Mean2026.06 | 71.5 | — | — | |
| MathstralPunctuation Percentage=30%2025.01 | 71.34 | — | — | |
| OriginalBlock size B=1282026.06 | 70.96 | — | — | |
| MathstralPunctuation Percentage=50%2025.01 | 70.65 | — | — | |
| DeepSeekPunctuation Percentage=30%2025.01 | 70.43 | — | — | |
| Before RL (Base)Backbone=Qwen2.5-3B-Base, Training Dataset=CodeContests, RL Strategy=None2026.05 | 67.93 | — | — | |
| DeepSeekPunctuation Percentage=50%2025.01 | 67.24 | — | — | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 67 | — | — | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 67 | — | 44.6 | |
| SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 67 | — | 61.53 | |
| DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 67 | — | 72.61 | |
| RainbowPaddingBlock size B=5122026.06 | 66.57 | — | — | |
| SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 66 | — | 59.81 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 66 | — | 69.49 | |
| RSRModel=Qwen2.5-7B2026.05 | 62.55 | — | — | |
| Qwen2.5Condition=Clean2025.01 | 61.1 | — | — | |
| RSRModel=Qwen2.5-3B2026.05 | 59.92 | — | — | |
| OriginalBlock size B=Mean2026.06 | 58.73 | — | — | |
| Qwen2.5Punctuation Percentage=10%2025.01 | 56.02 | — | — | |
| RSR-ItemModel=Qwen2.5-7B2026.05 | 55.12 | — | — | |
| HY-0.5B-FP16Model Scale=0.5B, Precision=FP16, Instruction-tuned=true2026.02 | 55.04 | — | — | |
| Qwen2.5Punctuation Percentage=30%2025.01 | 52.69 | — | — | |
| Gemma2Condition=Clean2025.01 | 49.65 | — | — | |
| Qwen2.5Punctuation Percentage=50%2025.01 | 49.35 | — | — | |
| Gemma2Punctuation Percentage=10%2025.01 | 45.1 | — | — | |
| SCASModel=Qwen2.5-0.5B2026.05 | 45.03 | — | — | |
| MistralCondition=Clean2025.01 | 42.07 | — | — | |
| MistralPunctuation Percentage=10%2025.01 | 41.62 | — | — | |
| MistralPunctuation Percentage=30%2025.01 | 37.75 | — | — | |
| GRACEModel=Qwen2.5-0.5B2026.05 | 37.65 | — | — | |
| Gemma2Punctuation Percentage=30%2025.01 | 36.46 | — | — | |
| MistralPunctuation Percentage=50%2025.01 | 36.39 | — | — | |
| Gemma2Punctuation Percentage=50%2025.01 | 35.63 | — | — | |
| RSR-ItemModel=Qwen2.5-3B2026.05 | 34.95 | — | — | |
| OriginalBlock size B=5122026.06 | 28.43 | — | — | |
| RSRModel=Qwen2.5-0.5B2026.05 | 26.26 | — | — | |
| InfluenceModel=Qwen2.5-3B2026.05 | 25.22 | — | — | |
| ZephyrCondition=Clean2025.01 | 23.27 | — | — | |
| RouterModel=Qwen2.5-3B2026.05 | 23.2 | — | — | |
| RouterModel=Qwen2.5-7B2026.05 | 21.99 | — | — | |
| PPLModel=Qwen2.5-7B2026.05 | 20.6 | — | — | |
| IFDModel=Qwen2.5-7B2026.05 | 20.32 | — | — | |
| InfluenceModel=Qwen2.5-7B2026.05 | 19.61 | — | — | |
| PPLModel=Qwen2.5-3B2026.05 | 18.88 | — | — | |
| IFDModel=Qwen2.5-3B2026.05 | 18.32 | — | — | |
| ZephyrPunctuation Percentage=10%2025.01 | 18.04 | — | — | |
| ZephyrPunctuation Percentage=30%2025.01 | 18.04 | — | — | |
| RSR-ItemModel=Qwen2.5-0.5B2026.05 | 16.1 | — | — |