End-to-end Throughput on LLaMA-2-7B-Chat
449Throughput (tokens/sec)CPE
Evaluation Results
| Method | Links | |
|---|---|---|
| CPEBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 449 | |
| CISBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 443 | |
| H2OBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 441 | |
| CPEBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 441 | |
| HShare-1BS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 439 | |
| CISBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 433 | |
| HShare-0BS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 430 | |
| DSBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 423 | |
| CPEBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 421 | |
| H2OBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 416 | |
| CISBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 415 | |
| CPEBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 412 | |
| HShare-1BS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 411 | |
| QuestBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 410 | |
| CISBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 402 | |
| HShare-0BS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 398 | |
| H2OBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 396 | |
| CPEBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 384 | |
| GPT-FastBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 374 | |
| HShare-1BS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 365 | |
| CISBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 364 | |
| CPEBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 362 | |
| DSBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 360 | |
| CISBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 352 | |
| HShare-0BS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 350 | |
| QuestBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 287 | |
| DSBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 286 | |
| CPEBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 244 | |
| H2OBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 240 | |
| CPEBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 238 | |
| CISBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 236 | |
| HShare-1BS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 235 | |
| H2OBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 234 | |
| CISBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 233 | |
| CPEBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 233 | |
| GPT-FastBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 233 | |
| HShare-0BS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 231 | |
| GPT-FastBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 228 | |
| QuestBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 228 | |
| DSBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 228 | |
| H2OBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 228 | |
| CPEBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 227 | |
| HShare-1BS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 226 | |
| CPEBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 225 | |
| CISBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 224 | |
| HShare-0BS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 222 | |
| CISBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02 | 220 | |
| CPEBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 219 | |
| HShare-1BS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02 | 217 | |
| CISBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 216 | |
| CISBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02 | 216 | |
| HShare-0BS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02 | 214 | |
| DSBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 213 | |
| QuestBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 206 | |
| DSBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 201 | |
| GPT-FastBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 188 | |
| QuestBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 175 | |
| QuestBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 152 | |
| GPT-FastBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 136 | |
| GPT-FastBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02 | 118 |