LLM Generation Throughput (End-to-end generation)
751Throughput (tokens/s)STAR-KV
Evaluation results
| Method | Links | ||
|---|---|---|---|
| STAR-KVContext Length=1K, Batch Size=1282026.06 | 751 | 3.01 | |
| STAR-KVContext Length=2K, Batch Size=642026.06 | 400.7 | 3.07 | |
| Pytorch SDPAContext Length=1K, Batch Size=162026.06 | 249.8 | — | |
| STAR-KVContext Length=4K, Batch Size=322026.06 | 212.2 | 3.07 | |
| Pytorch SDPAContext Length=2K, Batch Size=82026.06 | 130.4 | — | |
| STAR-KVContext Length=8K, Batch Size=162026.06 | 110.6 | 3.11 | |
| Pytorch SDPAContext Length=4K, Batch Size=42026.06 | 69 | — | |
| STAR-KVContext Length=16K, Batch Size=82026.06 | 56.6 | 3.14 | |
| Pytorch SDPAContext Length=8K, Batch Size=22026.06 | 35.6 | — | |
| Pytorch SDPAContext Length=16K, Batch Size=12026.06 | 18 | — |