End-to-end throughput on Cloud Inference Workloads
824.22Throughput (tokens/s)SpeContext
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SpeContextModel=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 824.22 | 32 | 18.09 | |
| SpeContextModel=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 690.59 | 32 | 24.89 | |
| SpeContextModel=Qwen3-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 592.39 | 32 | 17.54 | |
| SpeContextModel=DeepSeek-Distill-Llama-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 526.47 | 16 | 6.02 | |
| Full Attn (FlashInfer)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 490.04 | 16 | 10.75 | |
| SpeContextModel=Qwen3-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 424.92 | 32 | 22.03 | |
| Full Attn (FlashInfer)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 420.12 | 16 | 12.44 | |
| ShadowKVModel=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 366.74 | 16 | 8.05 | |
| SpeContextModel=DeepSeek-Distill-Llama-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 346.88 | 16 | 7.4 | |
| SpeContextModel=Qwen3-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 336.71 | 16 | 5.58 | |
| Full Attn (FlashInfer)Model=DeepSeek-Distill-Llama-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 320.41 | 8 | 3.65 | |
| Full Attn (FlashInfer)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 314.25 | 8 | 11.32 | |
| Full Attn (FlashInfer)Model=Qwen3-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 259.28 | 8 | 4.29 | |
| Full Attn (FlashInfer)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 254.92 | 8 | 13.22 | |
| ShadowKVModel=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 240.47 | 16 | 8.67 | |
| Full Attn (FlashInfer)Model=DeepSeek-Distill-Llama-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 222.06 | 8 | 4.74 | |
| SpeContextModel=Qwen3-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 210.75 | 16 | 6.47 | |
| ShadowKVModel=DeepSeek-Distill-Llama-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 168.06 | 32 | 1.92 | |
| Full Attn (FlashInfer)Model=Qwen3-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 156.92 | 6 | 4.81 | |
| Full Attn (Flash Attn)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 145.88 | 16 | 3.2 | |
| ShadowKVModel=DeepSeek-Distill-Llama-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 132.07 | 64 | 2.81 | |
| Full Attn (Flash Attn)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 129.67 | 16 | 3.83 | |
| Full Attn (Flash Attn)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 87.74 | 8 | 3.16 | |
| Full Attn (Flash Attn)Model=DeepSeek-Distill-Llama-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 87.71 | 8 | 1 | |
| Full Attn (Flash Attn)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 62.89 | 8 | 3.26 | |
| Full Attn (Flash Attn)Model=Qwen3-8B, Input sequence length=16k, Output sequence length=2k2025.11 | 60.31 | 8 | 1 | |
| Full Attn (Flash Attn)Model=DeepSeek-Distill-Llama-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 46.89 | 6 | 1 | |
| Full Attn (Eager)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 45.57 | 4 | 1 | |
| Full Attn (Eager)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=16k2025.11 | 33.77 | 4 | 1 | |
| Full Attn (Flash Attn)Model=Qwen3-8B, Input sequence length=32k, Output sequence length=2k2025.11 | 32.56 | 6 | 1 | |
| Full Attn (Eager)Model=DeepSeek-Distill-Llama-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 27.74 | 4 | 1 | |
| Full Attn (Eager)Model=Qwen3-8B, Input sequence length=2k, Output sequence length=32k2025.11 | 19.28 | 4 | 1 |