Peak decode throughput on SDAR-8B-Chat 16K context
1,364Throughput (tokens/s)HERALD
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| HERALDBatch size (B)=902026.06 | 1,364 | 1.81 | |
| DenseBatch size (B)=152026.06 | 754 | 1 | |
| MAGE-OffloadBatch size (B)=162026.06 | 238 | 0.32 | |
| InfiniGenBatch size (B)=162026.06 | 178 | 0.24 |