Conversational Question Answering on LoCoMo Multi-Hop
26F1 ScoreMemWeaver
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MemWeaverBackbone=GPT-4o-mini, Avg. Input Tokens=6722026.01 | 26 | 17.08 | 10.9 | |
| LoCoMoBackbone=GPT-4o-mini, Avg. Input Tokens=21,6252026.01 | 24.35 | 16.91 | 8.5 | |
| A-MemBackbone=GPT-4o-mini, Avg. Input Tokens=2,7312026.01 | 23.68 | 16.74 | 8.42 | |
| MemWeaverBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7342026.01 | 21.91 | 16.47 | 7.25 | |
| MemWeaverBackbone=Llama3.2-1B, Avg. Input Tokens=10002026.01 | 12.96 | 9.77 | 3.72 | |
| A-MemBackbone=Qwen2.5-1.5B, Avg. Input Tokens=2,5742026.01 | 12.57 | 9.94 | 5.17 | |
| MemWeaverBackbone=Llama3.2-3B, Avg. Input Tokens=10002026.01 | 11.86 | 13.07 | 6.94 | |
| A-MemBackbone=Llama3.2-1B, Avg. Input Tokens=26472026.01 | 11.03 | 9.21 | 3.35 | |
| LoCoMoBackbone=Llama3.2-1B, Avg. Input Tokens=22,3122026.01 | 10.37 | 8.1 | 2.85 | |
| LoCoMoBackbone=Qwen2.5-1.5B, Avg. Input Tokens=22,4242026.01 | 10.23 | 7.96 | 2.14 | |
| A-MemBackbone=Llama3.2-3B, Avg. Input Tokens=25082026.01 | 9.96 | 13.21 | 6.62 | |
| ReadAgentBackbone=GPT-4o-mini, Avg. Input Tokens=6432026.01 | 9.15 | 6.48 | 2.47 | |
| LoCoMoBackbone=Llama3.2-3B, Avg. Input Tokens=22,3122026.01 | 8.09 | 6.88 | 2.29 | |
| MemoryBankBackbone=Qwen2.5-1.5B, Avg. Input Tokens=1,5552026.01 | 6.63 | 5.02 | 1.77 | |
| ReadAgentBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7522026.01 | 6.61 | 4.93 | 0 | |
| MemoryBankBackbone=GPT-4o-mini, Avg. Input Tokens=1,5132026.01 | 6.1 | 4.52 | 1.57 | |
| ReadAgentBackbone=Llama3.2-1B, Avg. Input Tokens=6652026.01 | 5.96 | 5.12 | 0.53 | |
| MemoryBankBackbone=Llama3.2-3B, Avg. Input Tokens=1,5532026.01 | 5.04 | 3.34 | 1.15 | |
| MemoryBankBackbone=Llama3.2-1B, Avg. Input Tokens=1,5532026.01 | 4.8 | 3.36 | 1.12 | |
| ReadAgentBackbone=Llama3.2-3B, Avg. Input Tokens=4612026.01 | 2.47 | 1.78 | 2.47 |