Peak decode throughput on LLaDA mini 16B (32K context) 2.0
2,717Throughput (tokens/s)HERALD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HERALDBatch size (B)=892026.06 | 2,717 | 2.47 | |
| DenseBatch size (B)=172026.06 | 1,098 | 1 | |
| MAGE-OffloadBatch size (B)=562026.06 | 390 | 0.36 | |
| InfiniGenBatch size (B)=162026.06 | 340 | 0.31 |