Inference Speed Evaluation on generation 128 tokens
2,747.88Inference Time (ms)BlockPruner
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BlockPrunerBackbone=Llama2-7B, Pruning Ratio (%)=24.002024.06 | 2,747.88 | 1.47 | |
| ShortGPTBackbone=Llama2-7B, Pruning Ratio (%)=24.032024.06 | 3,094.36 | 1.31 | |
| SliceGPTBackbone=Llama2-7B, Pruning Ratio (%)=24.472024.06 | 3,226.68 | 1.25 | |
| BlockPrunerBackbone=Llama2-13B, Pruning Ratio (%)=21.052024.06 | 3,873.2 | 1.88 | |
| OriginalBackbone=Llama2-7B, Pruning Ratio (%)=0.002024.06 | 4,044.3 | 1 | |
| SliceGPTBackbone=Llama2-13B, Pruning Ratio (%)=21.522024.06 | 4,099.08 | 1.78 | |
| ShortGPTBackbone=Llama2-13B, Pruning Ratio (%)=21.932024.06 | 4,111.65 | 1.77 | |
| OriginalBackbone=Llama2-13B, Pruning Ratio (%)=0.002024.06 | 7,285.73 | 1 |