Text Generation on XSUM (test)
30.4Throughput (tokens/s)H2O
Evaluation results
| Method | Links | |
|---|---|---|
| H2OModel size=6.7B, Batch size=1, Offloading=GPU (G), Hardware=NVIDIA T4 GPU, KV cache budget=20%2023.06 | 30.4 | |
| AccelerateModel size=6.7B, Batch size=1, Offloading=GPU (G), Hardware=NVIDIA T4 GPU2023.06 | 11.98 | |
| FlexGenModel size=6.7B, Batch size=1, Offloading=GPU (G), Hardware=NVIDIA T4 GPU2023.06 | 10.8 | |
| H2OModel size=30B, Batch size=180, Offloading=CPU (C), Hardware=NVIDIA T4 GPU, KV cache budget=20%2023.06 | 6.7 | |
| DeepSpeedModel size=6.7B, Batch size=6, Offloading=CPU (C), Hardware=NVIDIA T4 GPU2023.06 | 3.52 | |
| FlexGenModel size=30B, Batch size=44, Offloading=CPU (C), Hardware=NVIDIA T4 GPU2023.06 | 3.29 | |
| DeepSpeedModel size=30B, Batch size=2, Offloading=CPU (C), Hardware=NVIDIA T4 GPU2023.06 | 0.31 | |
| AccelerateModel size=30B, Batch size=2, Offloading=CPU (C), Hardware=NVIDIA T4 GPU2023.06 | 0.23 |