Inference Throughput on H200 GPU
13,890Throughput (2k Input)Surefire-1B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Surefire-1Bnlayers=16, dmodel=2560, rmlp/attn=3.6, GQA=9, Nnon-embed=965M, Hardware=H200 GPUs, Serving framework=vLLM2025.10 | 13,890 | 11,283 | |
| LLaMA-3.2-1Bnlayers=16, dmodel=2048, rmlp/attn=4.8, GQA=4, Nnon-embed=973M, Hardware=H200 GPUs, Serving framework=vLLM2025.10 | 11,948 | 9,306 | |
| LLaMA-3.2-1B-HFnlayers=16, dmodel=2048, rmlp/attn=4.8, GQA=4, Nnon-embed=973M, Hardware=H200 GPUs, Serving framework=vLLM2025.10 | 11,948 | 9,306 | |
| Panda-1Bnlayers=16, dmodel=2560, rmlp/attn=1.067, GQA=4, Nnon-embed=975M, Hardware=H200 GPUs, Serving framework=vLLM2025.10 | 8,961 | 6,218 | |
| OLMo-2-1B-HFnlayers=16, dmodel=2048, rmlp/attn=3, GQA=1, Nnon-embed=1.074B, Hardware=H200 GPUs, Serving framework=vLLM2025.10 | 7,486 | — |