Mathematical Reasoning on GSM8K 20 (test)
1.39Speedup (x)SWIFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SWIFTModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.39 | 98 | |
| SDFPModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.37 | 95 | |
| SDFPModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.36 | 95 | |
| Lookahead DecodingModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.35 | — | |
| SWIFTModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.31 | 98 | |
| Lookahead DecodingModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.31 | — | |
| Lookahead DecodingModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.29 | — | |
| SDFPModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.27 | 92 | |
| SWIFTModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1.25 | 92 | |
| VANILLAModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1 | — | |
| VANILLAModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1 | — | |
| VANILLAModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 1 | — | |
| Parallel DecodingModel=LLaMA-2-13B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 0.99 | — | |
| Parallel DecodingModel=LLaMA-2-13B-Chat, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 0.97 | — | |
| Parallel DecodingModel=LLaMA-2-70B, Decoding Strategy=Greedy, Batch size=1, Shot configuration=5-shot, Maximum output length=642026.02 | 0.97 | — |