Long-term Memory and Reasoning on Grocery Shopping CSQA Source (train and dev)
81.44ACCcSirLLM
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| SirLLMLLM Backbone=Yi-34b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 81.44 | 89.6 | 0.09 | 63.31 | |
| StreamLLMLLM Backbone=Yi-34b, # Entropy=0, # Recent=1020, Attention Sink Size=4, Ndecay=12024.05 | 81.35 | 26.29 | — | — | |
| SirLLMLLM Backbone=Yi-6b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 72.44 | 99.27 | 1.11 | 73.54 | |
| StreamLLMLLM Backbone=Yi-6b, # Entropy=0, # Recent=1020, Attention Sink Size=4, Ndecay=12024.05 | 71.33 | 25.73 | — | — | |
| RandomLLMLLM Backbone=Yi-6b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 70.33 | 77.55 | -1 | 51.82 | |
| IntervalLLMLLM Backbone=Yi-6b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 63.98 | 21.72 | -7.2 | -4.01 | |
| SirLLMLLM Backbone=Vicuna-13b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 60.23 | 97.08 | 0.13 | 72.63 | |
| StreamLLMLLM Backbone=Vicuna-13b, # Entropy=0, # Recent=1020, Attention Sink Size=4, Ndecay=12024.05 | 60.1 | 24.45 | — | — | |
| SirLLMLLM Backbone=Vicuna-7b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 51.04 | 96.17 | 0.2 | 67.52 | |
| RandomLLMLLM Backbone=Vicuna-7b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 50.97 | 85.04 | 0.13 | 56.39 | |
| StreamLLMLLM Backbone=Vicuna-7b, # Entropy=0, # Recent=1020, Attention Sink Size=4, Ndecay=12024.05 | 50.84 | 28.65 | — | — | |
| IntervalLLMLLM Backbone=Vicuna-7b, # Entropy=1020, # Recent=0, Attention Sink Size=4, Ndecay=12024.05 | 47.21 | 23.72 | -3.63 | -4.93 |