# LLM Inference Acceleration on GSM8K

Source: [LLM Inference Acceleration on GSM8K](<https://www.sota2.com/research/sota/llm-inference-acceleration-on-gsm8k>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-27T10:00:23.262Z

Evaluation of model inference speedup and associated latency characteristics (measured by Tau) when running tasks on the GSM8K mathematical reasoning dataset.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | LLM Inference Acceleration |
| Dataset (including any stated split) | GSM8K |
| Metric | Speedup |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 62 of 62 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Speedup](<https://www.sota2.com/research/sota/llm-inference-acceleration-on-gsm8k.md?metric=Speedup>): Higher is better; unit: SCALAR.

- [Tau (τ)](<https://www.sota2.com/research/sota/llm-inference-acceleration-on-gsm8k.md?metric=Tau%20(%CF%84)>): Lower is better; unit: TIME.

- [TTFT (ms)](<https://www.sota2.com/research/sota/llm-inference-acceleration-on-gsm8k.md?metric=TTFT%20(ms)>): Lower is better; unit: TIME.

- [TPOT (ms)](<https://www.sota2.com/research/sota/llm-inference-acceleration-on-gsm8k.md?metric=TPOT%20(ms)>): Lower is better; unit: TIME.

## Recorded results

| Method | Speedup | Result date | Report paper |
| --- | --- | --- | --- |
| CATS w/ EAGLE | 5.51 | Not recorded | 6a040579dbc836b87d827444 |
| CATS w/ EAGLE | 5.48 | Not recorded | 6a040579dbc836b87d827444 |
| CATS w/ EAGLE | 5.42 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 5.27 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 5.19 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 5.1 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self, tree | 5.1 | Not recorded | 6a040579dbc836b87d827444 |
| FLy | 4.97 | Not recorded | 692d2169d6b0dd3342931b12 |
| CATS w/ EAGLE | 4.79 | Not recorded | 6a040579dbc836b87d827444 |
| FLy | 4.61 | Not recorded | 692d2169d6b0dd3342931b12 |
| Kangaroo | 4.59 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self | 4.57 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self, tree | 4.55 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self, tree | 4.39 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 4.31 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self, tree | 4.24 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self | 4.24 | Not recorded | 6a040579dbc836b87d827444 |
| Kangaroo | 4.23 | Not recorded | 6a040579dbc836b87d827444 |
| Kangaroo | 4.18 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self | 4.1 | Not recorded | 6a040579dbc836b87d827444 |
| CATS w/ EAGLE | 4.09 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self | 3.84 | Not recorded | 6a040579dbc836b87d827444 |
| Kangaroo | 3.82 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-3 | 3.72 | Not recorded | 692d2169d6b0dd3342931b12 |
| EAGLE-3 | 3.5 | Not recorded | 692d2169d6b0dd3342931b12 |
| CATS w/ EAGLE | 3.5 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self, tree | 3.33 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 3.05 | Not recorded | 6a040579dbc836b87d827444 |
| FLy | 2.98 | Not recorded | 692d2169d6b0dd3342931b12 |
| FLy | 2.94 | Not recorded | 692d2169d6b0dd3342931b12 |
| FLy | 2.91 | Not recorded | 692d2169d6b0dd3342931b12 |
| FLy | 2.85 | Not recorded | 692d2169d6b0dd3342931b12 |
| EAGLE-self, tree | 2.85 | Not recorded | 6a040579dbc836b87d827444 |
| Kangaroo | 2.76 | Not recorded | 6a040579dbc836b87d827444 |
| CATS | 2.74 | Not recorded | 6a040579dbc836b87d827444 |
| FLy | 2.68 | Not recorded | 692d2169d6b0dd3342931b12 |
| FLy | 2.67 | Not recorded | 692d2169d6b0dd3342931b12 |
| SpS | 2.59 | Not recorded | 692d2169d6b0dd3342931b12 |
| EAGLE-self | 2.58 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-2 | 2.54 | Not recorded | 692d2169d6b0dd3342931b12 |
| Kangaroo | 2.38 | Not recorded | 6a040579dbc836b87d827444 |
| Medusa | 2.38 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-2 | 2.36 | Not recorded | 692d2169d6b0dd3342931b12 |
| Medusa | 2.2 | Not recorded | 6a040579dbc836b87d827444 |
| EAGLE-self | 2.19 | Not recorded | 6a040579dbc836b87d827444 |
| Medusa | 2.19 | Not recorded | 6a040579dbc836b87d827444 |
| SpS | 2.18 | Not recorded | 692d2169d6b0dd3342931b12 |
| SpS | 2.13 | Not recorded | 692d2169d6b0dd3342931b12 |
| SpS | 2.13 | Not recorded | 692d2169d6b0dd3342931b12 |
| TokenRecycling | 2.1 | Not recorded | 692d2169d6b0dd3342931b12 |
| TokenRecycling | 2.08 | Not recorded | 692d2169d6b0dd3342931b12 |
| REST | 2.07 | Not recorded | 692d2169d6b0dd3342931b12 |
| Medusa | 1.92 | Not recorded | 6a040579dbc836b87d827444 |
| REST | 1.79 | Not recorded | 692d2169d6b0dd3342931b12 |
| REST | 1.78 | Not recorded | 692d2169d6b0dd3342931b12 |
| ReMoE | 1.77 | Not recorded | 6a167accdbd7150506f80219 |
| Medusa | 1.74 | Not recorded | 6a040579dbc836b87d827444 |
| REST | 1.71 | Not recorded | 692d2169d6b0dd3342931b12 |
| Medusa | 1.68 | Not recorded | 6a040579dbc836b87d827444 |
| TokenRecycling | 1.48 | Not recorded | 692d2169d6b0dd3342931b12 |
| TokenRecycling | 1.47 | Not recorded | 692d2169d6b0dd3342931b12 |

### CATS w/ EAGLE

Speedup: 5.51. Result ID: 6a0427b447f34ce924e9a92b.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### CATS w/ EAGLE

Speedup: 5.48. Result ID: 6a0427b447f34ce924e9a931.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### CATS w/ EAGLE

Speedup: 5.42. Result ID: 6a0427b447f34ce924e9a919.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### CATS

Speedup: 5.27. Result ID: 6a0427b447f34ce924e9a92f.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### CATS

Speedup: 5.19. Result ID: 6a0427b447f34ce924e9a929.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### CATS

Speedup: 5.1. Result ID: 6a0427b447f34ce924e9a917.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 5.1. Result ID: 6a0427b447f34ce924e9a930.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### FLy

Speedup: 4.97. Result ID: 6996d63f5698665ec422abe9.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### CATS w/ EAGLE

Speedup: 4.79. Result ID: 6a0427b447f34ce924e9a91f.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### FLy

Speedup: 4.61. Result ID: 6996d63f5698665ec422abdd.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 4.59. Result ID: 6a0427b447f34ce924e9a92c.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 4.57. Result ID: 6a0427b447f34ce924e9a92e.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 4.55. Result ID: 6a0427b447f34ce924e9a92a.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 4.39. Result ID: 6a0427b447f34ce924e9a918.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### CATS

Speedup: 4.31. Result ID: 6a0427b447f34ce924e9a91d.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 4.24. Result ID: 6a0427b447f34ce924e9a91e.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 4.24. Result ID: 6a0427b447f34ce924e9a928.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 4.23. Result ID: 6a0427b447f34ce924e9a91a.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 4.18. Result ID: 6a0427b447f34ce924e9a926.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 4.1. Result ID: 6a0427b447f34ce924e9a916.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### CATS w/ EAGLE

Speedup: 4.09. Result ID: 6a0427b447f34ce924e9a925.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 3.84. Result ID: 6a0427b447f34ce924e9a91c.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 3.82. Result ID: 6a0427b447f34ce924e9a914.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-3

Speedup: 3.72. Result ID: 6996d63f5698665ec422abd4.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa783c1e209131cea7758e.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3.3-Instruct-70B","Training-based":true,"Temperature":0}. This record does not assert independent reproduction.

### EAGLE-3

Speedup: 3.5. Result ID: 6996d63f5698665ec422abe0.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa783c1e209131cea7758e.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3.3-Instruct-70B","Training-based":true,"Temperature":1}. This record does not assert independent reproduction.

### CATS w/ EAGLE

Speedup: 3.5. Result ID: 6a0427b447f34ce924e9a913.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 3.33. Result ID: 6a0427b447f34ce924e9a924.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### CATS

Speedup: 3.05. Result ID: 6a0427b447f34ce924e9a923.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### FLy

Speedup: 2.98. Result ID: 6996d63f5698665ec422abd9.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### FLy

Speedup: 2.94. Result ID: 6996d63f5698665ec422abd3.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### FLy

Speedup: 2.91. Result ID: 6996d63f5698665ec422abdf.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### FLy

Speedup: 2.85. Result ID: 6996d63f5698665ec422abe1.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3.3-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### EAGLE-self, tree

Speedup: 2.85. Result ID: 6a0427b447f34ce924e9a912.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"tree base"}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 2.76. Result ID: 6a0427b447f34ce924e9a920.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### CATS

Speedup: 2.74. Result ID: 6a0427b447f34ce924e9a911.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### FLy

Speedup: 2.68. Result ID: 6996d63f5698665ec422abd5.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3.3-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### FLy

Speedup: 2.67. Result ID: 6996d63f5698665ec422abe5.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### SpS

Speedup: 2.59. Result ID: 6996d63f5698665ec422abda.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e7a1e209131cea40258.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 2.58. Result ID: 6a0427b447f34ce924e9a922.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-2

Speedup: 2.54. Result ID: 6996d63f5698665ec422abd2.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73ea1e209131cea5edff.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3-Instruct-70B","Training-based":true,"Temperature":0}. This record does not assert independent reproduction.

### Kangaroo

Speedup: 2.38. Result ID: 6a0427b447f34ce924e9a90e.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73081e209131cea59efc.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### Medusa

Speedup: 2.38. Result ID: 6a0427b447f34ce924e9a921.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-2

Speedup: 2.36. Result ID: 6996d63f5698665ec422abde.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73ea1e209131cea5edff.md>)

Recorded evaluation configuration: {"Model":"Meta-Llama-3-Instruct-70B","Training-based":true,"Temperature":1}. This record does not assert independent reproduction.

### Medusa

Speedup: 2.2. Result ID: 6a0427b447f34ce924e9a90f.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### EAGLE-self

Speedup: 2.19. Result ID: 6a0427b447f34ce924e9a910.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77391e209131cea71c7d.md>)

Recorded evaluation configuration: {"Model":"Vicuna-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### Medusa

Speedup: 2.19. Result ID: 6a0427b447f34ce924e9a927.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### SpS

Speedup: 2.18. Result ID: 6996d63f5698665ec422abe6.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e7a1e209131cea40258.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### SpS

Speedup: 2.13. Result ID: 6996d63f5698665ec422abd6.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e7a1e209131cea40258.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### SpS

Speedup: 2.13. Result ID: 6996d63f5698665ec422abe2.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e7a1e209131cea40258.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### TokenRecycling

Speedup: 2.1. Result ID: 6996d63f5698665ec422abd8.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c31e209131cea7a5d4.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### TokenRecycling

Speedup: 2.08. Result ID: 6996d63f5698665ec422abe4.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c31e209131cea7a5d4.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### REST

Speedup: 2.07. Result ID: 6996d63f5698665ec422abdb.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72cd1e209131cea588ee.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### Medusa

Speedup: 1.92. Result ID: 6a0427b447f34ce924e9a92d.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Relaxed (temperature = 0.7)","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### REST

Speedup: 1.79. Result ID: 6996d63f5698665ec422abd7.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72cd1e209131cea588ee.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

### REST

Speedup: 1.78. Result ID: 6996d63f5698665ec422abe7.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72cd1e209131cea588ee.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### ReMoE

Speedup: 1.77. Result ID: 6a16c0a3fbc373592c106672.

[Reporting paper](<https://www.sota2.com/research/paper/6a167accdbd7150506f80219.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a167accdbd7150506f80219.md>)

Recorded evaluation configuration: {"Hardware":"Jetson Orin NX 16GB","Backend":"llama.cpp","Number of parallel requests (-np)":"1","Sequence length (-n)":"128","Memory mapping (--mmap)":"true"}. This record does not assert independent reproduction.

### Medusa

Speedup: 1.74. Result ID: 6a0427b447f34ce924e9a915.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Llama2-7b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### REST

Speedup: 1.71. Result ID: 6996d63f5698665ec422abe3.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72cd1e209131cea588ee.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-70B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### Medusa

Speedup: 1.68. Result ID: 6a0427b447f34ce924e9a91b.

[Reporting paper](<https://www.sota2.com/research/paper/6a040579dbc836b87d827444.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b11e209131cea5df0f.md>)

Recorded evaluation configuration: {"Model":"Llama2-13b","Decoding Strategy":"Greedy","Decoding Structure":"chain base"}. This record does not assert independent reproduction.

### TokenRecycling

Speedup: 1.48. Result ID: 6996d63f5698665ec422abe8.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c31e209131cea7a5d4.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":1}. This record does not assert independent reproduction.

### TokenRecycling

Speedup: 1.47. Result ID: 6996d63f5698665ec422abdc.

[Reporting paper](<https://www.sota2.com/research/paper/692d2169d6b0dd3342931b12.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c31e209131cea7a5d4.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-Instruct-405B","Training-based":false,"Temperature":0}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
