LLM Generation on Synthetic
50.4Latency (s)H2O (20%)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| H2O (20%)Sequence length=7000+1024, Model size=30B, Batch size=1, GPU=NVIDIA A1002023.06 | 50.4 | — | |
| H2O (20%)Sequence length=2048+2048, Model size=6.7B, Batch size=24, GPU=NVIDIA A1002023.06 | 53.5 | 918.9 | |
| FlexGenSequence length=7000+1024, Model size=30B, Batch size=1, GPU=NVIDIA A1002023.06 | 57 | — | |
| FlexGenSequence length=2048+2048, Model size=6.7B, Batch size=24, GPU=NVIDIA A1002023.06 | 99.5 | 494.1 | |
| H2O (20%)Sequence length=5000+5000, Model size=13B, Batch size=4, GPU=NVIDIA A1002023.06 | 155.4 | — | |
| FlexGenSequence length=5000+5000, Model size=13B, Batch size=4, GPU=NVIDIA A1002023.06 | 214.2 | — | |
| H2O (20%)Sequence length=2048+2048, Model size=6.7B, Batch size=64, GPU=NVIDIA A1002023.06 | — | 1,161 |