Operational Efficiency and Success Rate on GSM8K (test)
2.66Speedup (x)CTC-drafter
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CTC-drafterBase Model=Vicuna-13B2024.11 | 2.66 | — | — | — | 3.53 | |
| CTC-drafterBase Model=Vicuna-7B2024.11 | 2.43 | — | — | — | 3.53 | |
| MedusaBase Model=Vicuna-7B2024.11 | 2.33 | — | — | — | 2.78 | |
| MedusaBase Model=Vicuna-13B2024.11 | 2.21 | — | — | — | 2.68 | |
| CTC-drafterBase Model=Vicuna-33B2024.11 | 2.16 | — | — | — | 3.4 | |
| MedusaBase Model=Vicuna-33B2024.11 | 2.1 | — | — | — | 2.46 | |
| GSDModel=LLaMA-2-70b2024.07 | 1.96 | 0.917 | 793.1 | 27.7 | — | |
| SSDModel=LLaMA-2-70b2024.07 | 1.85 | 0.795 | 629.9 | — | — | |
| TSDModel=LLaMA-2-70b2024.07 | 1.81 | 0.894 | 8,574.6 | 0 | — | |
| Self SSDModel=LLaMA-2-70b2024.07 | 1.37 | — | — | — | — | |
| VanillaBase Model=Vicuna-7B2024.11 | 1 | — | — | — | 1 | |
| VanillaBase Model=Vicuna-13B2024.11 | 1 | — | — | — | 1 | |
| VanillaBase Model=Vicuna-33B2024.11 | 1 | — | — | — | 1 |