Peak decode throughput on SDAR-8B-Chat 8K context
1,484Throughput (tokens/s)HERALD
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| HERALDBatch size (B)=1602026.06 | 1,484 | 1.34 | |
| DenseBatch size (B)=312026.06 | 1,107 | 1 | |
| MAGE-OffloadBatch size (B)=162026.06 | 414 | 0.37 | |
| InfiniGenBatch size (B)=162026.06 | 364 | 0.33 |