List your company
ResearchFormerly WizWand

End-to-end Throughput on LLaMA-2-7B-Chat

449Throughput (tokens/sec)CPE
104.76194.13283.5372.87Feb 9, 2026
Updated 8mo ago

Evaluation results

MethodLinks
CPEBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02449
CISBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02443
H2OBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02441
CPEBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02441
HShare-1BS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02439
CISBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02433
HShare-0BS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02430
DSBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02423
CPEBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02421
H2OBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02416
CISBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02415
CPEBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02412
HShare-1BS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02411
QuestBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02410
CISBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02402
HShare-0BS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02398
H2OBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02396
CPEBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02384
GPT-FastBS=16, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02374
HShare-1BS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02365
CISBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02364
CPEBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02362
DSBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02360
CISBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02352
HShare-0BS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02350
QuestBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02287
DSBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02286
CPEBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02244
H2OBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02240
CPEBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02238
CISBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02236
HShare-1BS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02235
H2OBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02234
CISBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02233
CPEBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02233
GPT-FastBS=16, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02233
HShare-0BS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02231
GPT-FastBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02228
QuestBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02228
DSBS=8, Seqlen=1k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02228
H2OBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02228
CPEBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02227
HShare-1BS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02226
CPEBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02225
CISBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02224
HShare-0BS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02222
CISBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=162026.02220
CPEBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02219
HShare-1BS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(1/2-1/2-1/2)2026.02217
CISBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02216
CISBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Block size (s)=82026.02216
HShare-0BS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A100, Configuration=HShare(3/4-3/4-1/2)2026.02214
DSBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02213
QuestBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02206
DSBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02201
GPT-FastBS=8, Seqlen=2k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02188
QuestBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02175
QuestBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02152
GPT-FastBS=16, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02136
GPT-FastBS=8, Seqlen=4k, Backbone=LLaMA-2-7B-Chat, Hardware=NVIDIA A1002026.02118