Speculative Decoding Efficiency on CNN/DM, GSM8K, TinyStories Aggregate
28.26Decoding Speed (tokens/s)SWIFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SWIFTModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=12026.02 | 28.26 | 1.41 | |
| SDFPModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=12026.02 | 28.05 | 1.4 | |
| SWIFTModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=12026.02 | 26.8 | 1.34 | |
| SDFPModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=12026.02 | 26.21 | 1.31 | |
| Lookahead DecodingModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=12026.02 | 25.69 | 1.29 | |
| Lookahead DecodingModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=12026.02 | 25.46 | 1.27 | |
| VANILLAModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=12026.02 | 20.1 | 1 | |
| VANILLAModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=12026.02 | 19.96 | 1 | |
| Parallel DecodingModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=12026.02 | 19.49 | 0.97 | |
| Parallel DecodingModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=12026.02 | 19.26 | 0.97 | |
| SWIFTModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=12026.02 | 6.41 | 1.48 | |
| SDFPModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=12026.02 | 6.29 | 1.46 | |
| Lookahead DecodingModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=12026.02 | 5.45 | 1.26 | |
| VANILLAModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=12026.02 | 4.32 | 1 | |
| Parallel DecodingModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=12026.02 | 4.14 | 0.96 |