List your company
ResearchFormerly WizWand

Mathematical Reasoning on GSM8K (Speedup)

2.91SpeedupSlimSpec
1.31881.73192.1452.5581May 11, 2026
Updated 5mo ago

Evaluation results

MethodLinks
SlimSpecTarget Model=Llama-3.1-8B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.91
SpecVocabTarget Model=Llama-3.1-8B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.82
VocabTrim-TTarget Model=Llama-3.1-8B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.66
Full VocabTarget Model=Llama-3.1-8B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.42
SlimSpecTarget Model=Qwen3-30B-A3B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.27
SpecVocabTarget Model=Qwen3-30B-A3B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.24
VocabTrim-TTarget Model=Qwen3-30B-A3B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.23
Full VocabTarget Model=Qwen3-30B-A3B-Instruct, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.12
SlimSpecTarget Model=GPT-OSS-20B, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.052.11
SpecVocabTarget Model=GPT-OSS-20B, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.98
VocabTrim-TTarget Model=GPT-OSS-20B, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.95
Full VocabTarget Model=GPT-OSS-20B, Batch size=1, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.63
SlimSpecTarget Model=Llama-3.1-8B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.56
SlimSpecTarget Model=GPT-OSS-20B, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.56
SlimSpecTarget Model=Qwen3-30B-A3B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.54
VocabTrim-TTarget Model=Qwen3-30B-A3B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.53
SpecVocabTarget Model=Llama-3.1-8B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.52
VocabTrim-TTarget Model=GPT-OSS-20B, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.52
SpecVocabTarget Model=Qwen3-30B-A3B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.49
Full VocabTarget Model=Qwen3-30B-A3B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.48
VocabTrim-TTarget Model=Llama-3.1-8B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.46
SpecVocabTarget Model=GPT-OSS-20B, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.45
Full VocabTarget Model=Llama-3.1-8B-Instruct, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.41
Full VocabTarget Model=GPT-OSS-20B, Batch size=64, Temperature=0, Inference Framework=vLLM 0.17.1, Hardware=NVIDIA H2002026.051.38