Long Text Tasks on LongBench 4k-length (test)
8PR (Zh)Full Cache
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Full CacheBackbone=LLaMA2-7B-Chat, KV Cache Budget=100%, Sequence Length=4k2024.08 | 8 | 10.1 | 52.3 | 27.7 | 18.6 | 83.3 | 20.6 | 31.5 | 0 | |
| NACLBackbone=LLaMA2-7B-Chat, KV Cache Budget=20%, Sequence Length=4k2024.08 | 7 | 9.4 | 51.6 | 27.2 | 17.1 | 82.5 | 20.8 | 30.8 | -0.7 | |
| NACLBackbone=LLaMA2-7B-Chat, KV Cache Budget=30%, Sequence Length=4k2024.08 | 6.8 | 9 | 52.5 | 27.9 | 16.4 | 83.1 | 21.5 | 31 | -0.5 | |
| NACLBackbone=LLaMA2-7B-Chat, KV Cache Budget=10%, Sequence Length=4k2024.08 | 6.8 | 7 | 49.2 | 25.5 | 15 | 81.7 | 20.4 | 29.4 | -2.1 | |
| MSRNNBackbone=LLaMA2-7B-Chat, KV Cache Budget=30%, Sequence Length=4k2024.08 | 5.5 | 4.5 | 50.3 | 26.1 | 16.4 | 82.9 | 20.9 | 29.5 | -2 | |
| H2OBackbone=LLaMA2-7B-Chat, KV Cache Budget=10%, Sequence Length=4k2024.08 | 4.9 | 3.5 | 48.1 | 22.8 | — | 79.4 | 19.6 | 27.4 | -4.1 | |
| MSRNNBackbone=LLaMA2-7B-Chat, KV Cache Budget=20%, Sequence Length=4k2024.08 | 4.5 | 4.5 | 49 | 23.9 | 14.7 | 82.5 | 20.4 | 28.5 | -3 | |
| H2OBackbone=LLaMA2-7B-Chat, KV Cache Budget=20%, Sequence Length=4k2024.08 | 4.2 | 4.5 | 49.4 | 24.5 | 15.2 | 82.8 | 19.8 | 28.6 | -2.9 | |
| ScissorhandsBackbone=LLaMA2-7B-Chat, KV Cache Budget=30%, Sequence Length=4k2024.08 | 4 | 2.6 | 51.8 | 17.2 | 12.5 | 79.2 | 19.6 | 26.7 | -4.8 | |
| MSRNNBackbone=LLaMA2-7B-Chat, KV Cache Budget=10%, Sequence Length=4k2024.08 | 4 | 3 | 47.9 | 23.4 | 13.4 | 80.8 | 19.6 | 27.5 | -4 | |
| H2OBackbone=LLaMA2-7B-Chat, KV Cache Budget=30%, Sequence Length=4k2024.08 | 3.7 | 5 | 50.9 | 27.1 | 15.5 | 81.6 | 20.2 | 29.1 | -2.4 | |
| ScissorhandsBackbone=LLaMA2-7B-Chat, KV Cache Budget=20%, Sequence Length=4k2024.08 | 0.5 | 4.3 | 44.9 | 11.4 | 6.7 | 68.9 | 16.3 | 21.9 | -9.6 | |
| ScissorhandsBackbone=LLaMA2-7B-Chat, KV Cache Budget=10%, Sequence Length=4k2024.08 | 0 | 3.7 | 27.6 | 4.4 | 2.8 | 52.8 | 12.9 | 14.9 | -16.6 |