Inference Efficiency on Alpaca
102.44Latency (ms/tok)Baseline
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BaselineBackbone=Llama3-8B-Instruct, Sequence length=512, Batch size=12025.08 | 102.44 | 9.76 | 16.1 | |
| CAABackbone=Llama3-8B-Instruct, Sequence length=512, Batch size=12025.08 | 110.59 | 9.04 | 16.12 | |
| ReFTBackbone=Llama3-8B-Instruct, Sequence length=512, Batch size=12025.08 | 115.72 | 8.64 | 16.26 | |
| MSRSBackbone=Llama3-8B-Instruct, Sequence length=512, Batch size=12025.08 | 120.35 | 8.38 | 16.29 |