Peak decode throughput on SDAR-8B-Chat 32K context
823Throughput (tokens/s)HERALD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HERALDBatch size (B)=502026.06 | 823 | 1.92 | |
| DenseBatch size (B)=72026.06 | 429 | 1 | |
| MAGE-OffloadBatch size (B)=562026.06 | 121 | 0.28 | |
| InfiniGenBatch size (B)=162026.06 | 89 | 0.21 |