LLM Decoding Throughput on LLaDA mini 16B 8K context 2.0
3,813Throughput (tokens/s)HERALD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HERALDBatch size (B)=2002026.06 | 3,813 | 1.4 | |
| DenseBatch size (B)=702026.06 | 2,712 | 1 | |
| MAGE-OffloadBatch size (B)=1282026.06 | 1,369 | 0.5 | |
| InfiniGenBatch size (B)=322026.06 | 867 | 0.32 |