Speculative Decoding on HumanEval (test)
8.02BESpecTr-GBV
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SpecTr-GBVTarget Model=CodeLlama-13B, Draft Model=CodeLlama-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 8.02 | 1.67 | |
| SpecTrTarget Model=CodeLlama-13B, Draft Model=CodeLlama-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 7.99 | 1.67 | |
| GBVTarget Model=CodeLlama-13B, Draft Model=CodeLlama-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 7.59 | 1.42 | |
| SDTarget Model=CodeLlama-13B, Draft Model=CodeLlama-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 7.58 | 1.45 | |
| SpecTr-GBVTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 6.96 | 1.31 | |
| SpecTrTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 6.88 | 1.2 | |
| GBVTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 6.5 | 1.15 | |
| SDTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 6.44 | 1.18 | |
| ARTarget Model=CodeLlama-13B, Draft Model=CodeLlama-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 1 | 1 | |
| ARTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Draft length (L)=8, Temperature (T)=0.4, Draft number (K)=32026.04 | 1 | 1 |