Mathematical Reasoning on GSM8K (test) (Latency, Throughput)
52Latency (min)SkipKV
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| SkipKVBatch-size=100, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 52 | 25.4 | |
| SkipKVBatch-size=50, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 58 | 22.7 | |
| R-KVBatch-size=100, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 66 | 20 | |
| SkipKVBatch-size=120, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 66 | 20 | |
| SkipKVBatch-size=140, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 68 | 19.4 | |
| R-KVBatch-size=140, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 70 | 18.8 | |
| R-KVBatch-size=120, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 73 | 18.1 | |
| R-KVBatch-size=50, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 96 | 13.7 | |
| SkipKVBatch-size=10, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 136 | 9.7 | |
| SEALBatch-size=10, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, Steering factor=1, Attention Mechanism=FlashAttention-22025.12 | 178 | 7.41 | |
| SEALBatch-size=50, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, Steering factor=1, Attention Mechanism=FlashAttention-22025.12 | 192 | 6.87 | |
| R-KVBatch-size=10, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, KV-cache budget=512, Attention Mechanism=FlashAttention-22025.12 | 227 | 5.81 | |
| FullKVBatch-size=10, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, Attention Mechanism=FlashAttention-22025.12 | 324 | 4.07 | |
| FullKVBatch-size=50, Model=R1-Qwen-7B, Hardware=A100-40GB, Maximum generation length=8,192 tokens, Attention Mechanism=FlashAttention-22025.12 | 500 | 2.64 |