Generation throughput on Synthetic Generation Workload 512 prompt + 1024 generation tokens
52.1Throughput (tokens/s)H2O
Evaluation Results
| Method | Links | |
|---|---|---|
| H2OModel size=6.7B, Effective batch size=4, Memory hierarchy level=GPU, KV cache budget=20%2023.06 | 52.1 | |
| FlexGenModel size=6.7B, Effective batch size=1, Memory hierarchy level=GPU2023.06 | 16.9 | |
| H2OModel size=30B, Effective batch size=264, Memory hierarchy level=CPU, KV cache budget=20%2023.06 | 13.82 | |
| DeepSpeedModel size=6.7B, Effective batch size=16, Memory hierarchy level=CPU2023.06 | 10.1 | |
| FlexGenModel size=30B, Effective batch size=48, Memory hierarchy level=CPU2023.06 | 7.1 | |
| AccelerateModel size=6.7B, Effective batch size=16, Memory hierarchy level=CPU2023.06 | 5.6 | |
| AccelerateModel size=30B, Effective batch size=8, Memory hierarchy level=CPU2023.06 | 0.6 | |
| DeepSpeedModel size=30B, Effective batch size=4, Memory hierarchy level=CPU2023.06 | 0.6 |