Generation throughput on Synthetic Generation Workload 512 prompt + 512 generation tokens
51.7Throughput (tokens/s)H2O
Evaluation Results
| Method | Links | |
|---|---|---|
| H2OModel size=6.7B, Effective batch size=4, Memory hierarchy level=GPU, KV cache budget=20%2023.06 | 51.7 | |
| H2OModel size=30B, Effective batch size=416, Memory hierarchy level=CPU, KV cache budget=20%2023.06 | 18.83 | |
| FlexGenModel size=6.7B, Effective batch size=1, Memory hierarchy level=GPU2023.06 | 16.8 | |
| AccelerateModel size=6.7B, Effective batch size=1, Memory hierarchy level=GPU2023.06 | 15.5 | |
| DeepSpeedModel size=6.7B, Effective batch size=16, Memory hierarchy level=CPU2023.06 | 9.6 | |
| FlexGenModel size=30B, Effective batch size=80, Memory hierarchy level=CPU2023.06 | 8.5 | |
| AccelerateModel size=30B, Effective batch size=8, Memory hierarchy level=CPU2023.06 | 0.6 | |
| DeepSpeedModel size=30B, Effective batch size=4, Memory hierarchy level=CPU2023.06 | 0.6 |