Inference Efficiency on KV Cache Efficiency
1,966SKV CountL2-7B (Base)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| L2-7B (Base)Model=L2-7B, dqk=128, dvo=128, Method=Base2025.07 | 1,966 | — | 668 | — | |
| KV-Latent Train (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Train2025.07 | 983 | 50 | 573 | 17 | |
| KV-Latent Distill (L2-7B)Model=L2-7B, dqk=64, dvo=64, Method=Distill2025.07 | 983 | 50 | 573 | 17 | |
| L3-8B (Base)Model=L3-8B, dqk=128, dvo=128, Method=Base2025.07 | 491 | — | 670 | — | |
| KV-Latent Train (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Train2025.07 | 245 | 50 | 622 | 8 | |
| KV-Latent Distill (L3-8B)Model=L3-8B, dqk=64, dvo=64, Method=Distill2025.07 | 245 | 50 | 622 | 8 | |
| KV-Latent Train (L3-8B, d=16)Model=L3-8B, dqk=16, dvo=16, Method=Train2025.07 | 64 | 87 | 595 | 13 |