Generation throughput on Synthetic Generation Workload 512 prompt + 32 generation tokens
35.1Throughput (tokens/s)H2O
Evaluation Results
| Method | Links | |
|---|---|---|
| H2OModel size=6.7B, Effective batch size=4, Memory hierarchy level=GPU, KV cache budget=20%2023.06 | 35.1 | |
| AccelerateModel size=6.7B, Effective batch size=2, Memory hierarchy level=GPU2023.06 | 20.4 | |
| FlexGenModel size=6.7B, Effective batch size=2, Memory hierarchy level=GPU2023.06 | 20.2 | |
| H2OModel size=30B, Effective batch size=728, Memory hierarchy level=CPU, KV cache budget=20%2023.06 | 12.7 | |
| DeepSpeedModel size=6.7B, Effective batch size=16, Memory hierarchy level=CPU2023.06 | 10.2 | |
| FlexGenModel size=30B, Effective batch size=144, Memory hierarchy level=CPU2023.06 | 8.1 | |
| AccelerateModel size=30B, Effective batch size=8, Memory hierarchy level=CPU2023.06 | 0.6 | |
| DeepSpeedModel size=30B, Effective batch size=4, Memory hierarchy level=CPU2023.06 | 0.6 |