Efficiency Benchmarking on Qwen3-8B (Single-layer forward+backward setup)
36.6Time (ms)DualKV
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| DualKVPrompt length (P)=5K, Micro-batch size (mb)=42026.05 | 36.6 | 2.55 | 1.98 | 4 | |
| DualKVPrompt length (P)=8K, Micro-batch size (mb)=42026.05 | 57.2 | 2.6 | 2.17 | 4.9 | |
| DualKVPrompt length (P)=5K, Micro-batch size (mb)=82026.05 | 61.4 | 2.5 | 2.55 | 5.5 | |
| DualKVPrompt length (P)=8K, Micro-batch size (mb)=82026.05 | 70.3 | 3.49 | 3.3 | 6.5 | |
| Prefix Grouper (PG)Prompt length (P)=5K, Micro-batch size (mb)=42026.05 | 72.5 | — | — | 9 | |
| FA2Prompt length (P)=5K, Micro-batch size (mb)=42026.05 | 93.2 | — | — | 4 | |
| DualKVPrompt length (P)=16K, Micro-batch size (mb)=42026.05 | 108.9 | 3.03 | 2.26 | 7.5 | |
| DualKVPrompt length (P)=5K, Micro-batch size (mb)=162026.05 | 110.4 | 3.07 | 2.85 | 9 | |
| Prefix Grouper (PG)Prompt length (P)=8K, Micro-batch size (mb)=42026.05 | 124.3 | — | — | 12.8 | |
| DualKVPrompt length (P)=16K, mb=8, Hardware=H1002026.05 | 125 | 4.41 | 3.47 | 9 | |
| DualKVPrompt length (P)=16K, Micro-batch size (mb)=82026.05 | 125.4 | 4.41 | 3.47 | 9 | |
| DualKVPrompt length (P)=8K, mb=16, Hardware=H1002026.05 | 128 | 4.28 | 3.63 | 9.9 | |
| DualKVPrompt length (P)=8K, Micro-batch size (mb)=162026.05 | 128.2 | 4.28 | 3.63 | 9.9 | |
| FA2Prompt length (P)=8K, Micro-batch size (mb)=42026.05 | 148.9 | — | — | 4.9 | |
| FA2Prompt length (P)=5K, Micro-batch size (mb)=82026.05 | 153.3 | — | — | 5.5 | |
| Prefix Grouper (PG)Prompt length (P)=5K, Micro-batch size (mb)=82026.05 | 156.9 | — | — | 18.1 | |
| DualKVPrompt length (P)=5K, Micro-batch size (mb)=322026.05 | 179.9 | 3.32 | 3.48 | 13.7 | |
| DualKVPrompt length (P)=5K, mb=32, Hardware=H1002026.05 | 180 | 3.32 | 3.48 | 13.7 | |
| DualKVPrompt length (P)=8K, Micro-batch size (mb)=322026.05 | 184.2 | — | — | 14.6 | |
| DualKVPrompt length (P)=16K, Micro-batch size (mb)=162026.05 | 205.8 | — | — | 12.5 | |
| DualKVPrompt length (P)=32K, Micro-batch size (mb)=22026.05 | 220.3 | 1.85 | 1.4 | 11.8 | |
| Prefix Grouper (PG)Prompt length (P)=8K, Micro-batch size (mb)=82026.05 | 232.1 | — | — | 25.8 | |
| FA2Prompt length (P)=8K, Micro-batch size (mb)=82026.05 | 245.4 | — | — | 6.5 | |
| Prefix Grouper (PG)Prompt length (P)=16K, Micro-batch size (mb)=42026.05 | 245.7 | — | — | 22.9 | |
| DualKVPrompt length (P)=32K, mb=4, Hardware=H1002026.05 | 246 | 3.29 | 2.08 | 12.5 | |
| DualKVPrompt length (P)=32K, Micro-batch size (mb)=42026.05 | 246.4 | 3.29 | 2.08 | 12.5 | |
| DualKVPrompt length (P)=32K, Micro-batch size (mb)=82026.05 | 266.8 | — | — | 14.1 | |
| Prefix Grouper (PG)Prompt length (P)=32K, Micro-batch size (mb)=22026.05 | 308.1 | — | — | 22.1 | |
| Prefix Grouper (PG)Prompt length (P)=5K, Micro-batch size (mb)=162026.05 | 314.2 | — | — | 35.4 | |
| FA2Prompt length (P)=16K, Micro-batch size (mb)=42026.05 | 329.9 | — | — | 7.5 | |
| FA2Prompt length (P)=5K, Micro-batch size (mb)=162026.05 | 338.9 | — | — | 9 | |
| FA2Prompt length (P)=32K, Micro-batch size (mb)=22026.05 | 408 | — | — | 11.8 | |
| Prefix Grouper (PG)Prompt length (P)=16K, Micro-batch size (mb)=82026.05 | 434.9 | — | — | 45.9 | |
| PGPrompt length (P)=16K, mb=8, Hardware=H1002026.05 | 435 | — | — | 45.9 | |
| PGPrompt length (P)=8K, mb=16, Hardware=H1002026.05 | 465 | — | — | 50.7 | |
| Prefix Grouper (PG)Prompt length (P)=8K, Micro-batch size (mb)=162026.05 | 465.4 | — | — | 50.7 | |
| Prefix Grouper (PG)Prompt length (P)=32K, Micro-batch size (mb)=42026.05 | 513.6 | — | — | 43 | |
| PGPrompt length (P)=32K, mb=4, Hardware=H1002026.05 | 514 | — | — | 43 | |
| FA2Prompt length (P)=8K, mb=16, Hardware=H1002026.05 | 549 | — | — | — | |
| FA2Prompt length (P)=8K, Micro-batch size (mb)=162026.05 | 549.2 | — | — | 9.9 | |
| FA2Prompt length (P)=16K, Micro-batch size (mb)=82026.05 | 553 | — | — | 9 | |
| FA2Prompt length (P)=16K, mb=8, Hardware=H1002026.05 | 553 | — | — | — | |
| FA2Prompt length (P)=5K, mb=32, Hardware=H1002026.05 | 597 | — | — | — | |
| FA2Prompt length (P)=5K, Micro-batch size (mb)=322026.05 | 597.2 | — | — | 13.7 | |
| DualKVPrompt length (P)=65K, Micro-batch size (mb)=22026.05 | 614.1 | 1.91 | 1.26 | 21.9 | |
| PGPrompt length (P)=5K, mb=32, Hardware=H1002026.05 | 626 | — | — | 69.8 | |
| Prefix Grouper (PG)Prompt length (P)=5K, Micro-batch size (mb)=322026.05 | 626.3 | — | — | 69.8 | |
| DualKVPrompt length (P)=65K, Micro-batch size (mb)=42026.05 | 653.9 | — | — | 22.7 | |
| DualKVPrompt length (P)=65K, mb=4, Hardware=H1002026.05 | 654 | — | — | 22.7 | |
| Prefix Grouper (PG)Prompt length (P)=65K, Micro-batch size (mb)=22026.05 | 773.3 | — | — | 42.3 | |
| FA2Prompt length (P)=32K, mb=4, Hardware=H1002026.05 | 810 | — | — | — | |
| FA2Prompt length (P)=32K, Micro-batch size (mb)=42026.05 | 810.2 | — | — | 12.5 | |
| DualKVPrompt length (P)=65K, Micro-batch size (mb)=162026.05 | 892.6 | — | — | 27.7 | |
| FA2Prompt length (P)=65K, Micro-batch size (mb)=22026.05 | 1,173.5 | — | — | 21.9 | |
| DualKVPrompt length (P)=131K, Micro-batch size (mb)=82026.05 | 2,096.7 | — | — | 44.5 | |
| DualKVPrompt length (P)=131K, mb=8, Hardware=H1002026.05 | 2,097 | — | — | 44.5 | |
| DualKVPrompt length (P)=131K, Micro-batch size (mb)=322026.05 | 2,883 | — | — | 52.6 |