Conversational Question Answering on LoCoMo Single-Hop
42.39F1 ScoreLoCoMo
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LoCoMoBackbone=GPT-4o-mini, Avg. Input Tokens=21,6252026.01 | 42.39 | 31.65 | 28.29 | |
| MemWeaverBackbone=GPT-4o-mini, Avg. Input Tokens=6722026.01 | 39.2 | 33.58 | 25.96 | |
| A-MemBackbone=GPT-4o-mini, Avg. Input Tokens=2,7312026.01 | 35.13 | 28.96 | 21.53 | |
| MemWeaverBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7342026.01 | 32.8 | 27.42 | 18.52 | |
| MemWeaverBackbone=Llama3.2-3B, Avg. Input Tokens=10002026.01 | 21.22 | 22.27 | 20.28 | |
| MemWeaverBackbone=Llama3.2-1B, Avg. Input Tokens=10002026.01 | 20.17 | 15.05 | 11.78 | |
| A-MemBackbone=Qwen2.5-1.5B, Avg. Input Tokens=2,5742026.01 | 20.06 | 14.86 | 11.07 | |
| A-MemBackbone=Llama3.2-3B, Avg. Input Tokens=25082026.01 | 19.85 | 19.7 | 16.98 | |
| A-MemBackbone=Llama3.2-1B, Avg. Input Tokens=26472026.01 | 19.2 | 15.01 | 7.63 | |
| LoCoMoBackbone=Llama3.2-1B, Avg. Input Tokens=22,3122026.01 | 13.9 | 10.64 | 4.88 | |
| LoCoMoBackbone=Qwen2.5-1.5B, Avg. Input Tokens=22,4242026.01 | 12.93 | 9.18 | 4.14 | |
| LoCoMoBackbone=Llama3.2-3B, Avg. Input Tokens=22,3122026.01 | 11.92 | 8.63 | 5.02 | |
| ReadAgentBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7522026.01 | 10.13 | 7.54 | 0 | |
| ReadAgentBackbone=GPT-4o-mini, Avg. Input Tokens=6432026.01 | 9.67 | 7.66 | 2.99 | |
| MemoryBankBackbone=Qwen2.5-1.5B, Avg. Input Tokens=1,5552026.01 | 9.41 | 4.85 | 3.38 | |
| MemoryBankBackbone=GPT-4o-mini, Avg. Input Tokens=1,5132026.01 | 8.42 | 3.96 | 3.21 | |
| ReadAgentBackbone=Llama3.2-1B, Avg. Input Tokens=6652026.01 | 7.75 | 6.03 | 1.19 | |
| MemoryBankBackbone=Llama3.2-3B, Avg. Input Tokens=1,5532026.01 | 6.78 | 2.9 | 2.34 | |
| MemoryBankBackbone=Llama3.2-1B, Avg. Input Tokens=1,5532026.01 | 6.42 | 3.02 | 2.15 | |
| ReadAgentBackbone=Llama3.2-3B, Avg. Input Tokens=4612026.01 | 3.25 | 2.51 | 3.25 |