Inference Efficiency on 1x H100 96GB GPU (synthetic)
161,312ThroughputSRM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SRMImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=10242026.05 | 161,312 | 512,000 | |
| MambaImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=2562026.05 | 61,465 | 1,000 | |
| TransformerImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=2562026.05 | 32,134 | 6,000 | |
| MambaImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=5122026.05 | 23,155 | 1,000 | |
| TransformerImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=5122026.05 | 15,272 | 4,000 | |
| RWKVImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=2562026.05 | 3,820 | 1,024 | |
| RWKVImplementation=Pytorch, Context Length (n_ctx)=512, Number of Layers (n_l)=8, Model Dimension (d_m)=5122026.05 | 3,774 | 1,024 |