Natural Language Generation on EverythingLM
35.4BLEUFull Attention
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full AttentionBackbone=Llama-2 7B, Context length=4,0002024.07 | 35.4 | 30.8 | 64.5 | |
| ADOREBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 30.4 | 27.7 | 63.1 | |
| H2O(Rebuilt)Backbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 25.2 | 25.6 | 62.1 | |
| H2OBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 24.6 | 25.1 | 61.8 | |
| Window AttentionBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 22.3 | 25.9 | 62.3 | |
| StreamingLLMBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 20.5 | 25.6 | 61.4 | |
| Strided AttentionBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 20.3 | 22.1 | 58.5 | |
| KV CompressionBackbone=Llama-2 7B, Context length=4,000, KV cache size=1922024.07 | 18.2 | 22.3 | 55.7 |