List your company
ResearchFormerly WizWand

LLM Inference Acceleration on GSM8K

5.51SpeedupCATS w/ EAGLE
1.30842.39923.494.5808Nov 28, 2025Dec 27, 2025Jan 26, 2026Feb 25, 2026Mar 27, 2026Apr 26, 2026May 26, 2026
Updated 5mo ago

Evaluation results

MethodLinks
CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.055.515.5096——
CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.055.485.5524——
CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.055.425.4185——
CATSModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.055.275.3102——
CATSModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.055.195.187——
CATSModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.055.15.1055——
EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.055.15.1449——
FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.114.9719.4——
CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.054.794.8534——
FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.114.6117.02——
KangarooModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.054.594.6031——
EAGLE-selfModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.054.574.5883——
EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.054.555.2814——
EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.054.395.0954——
CATSModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.054.314.3415——
EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.054.244.2793——
EAGLE-selfModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.054.244.9139——
KangarooModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.054.234.2421——
KangarooModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.054.184.8506——
EAGLE-selfModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.054.14.7597——
CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.054.094.1399——
EAGLE-selfModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.053.843.8572——
KangarooModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.053.824.4296——
EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=02025.113.725.64——
EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=12025.113.55.31——
CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.053.53.5481——
EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.053.333.3631——
CATSModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.053.053.0736——
FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.112.9812.57——
FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=02025.112.9411.67——
FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=12025.112.9111.55——
FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=12025.112.8512.4——
EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.052.852.876——
KangarooModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.052.762.7695——
CATSModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.052.742.7636——
FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=02025.112.6811.65——
FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.112.6711.93——
SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.112.5910.98——
EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.052.582.5893——
EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=02025.112.543.74——
KangarooModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.052.382.3848——
MedusaModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.052.382.792——
EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=12025.112.363.48——
MedusaModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.052.22.5824——
EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.052.192.2031——
MedusaModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.052.192.1875——
SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.112.189.25——
SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.112.139.87——
SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.112.139.7——
TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.112.13.04——
TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.112.083.02——
RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.112.072.13——
MedusaModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.051.922.2499——
RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.111.792——
RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.111.781.82——
ReMoEHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.051.77—4,736.7346.04
MedusaModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.051.741.7358——
RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.111.711.91——
MedusaModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.051.681.9721——
TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.111.482.85——
TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.111.472.83——
BaselineHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.05——6,041.65613.73