Conversational Question Answering on LoCoMo Temporal
50.83F1 ScoreMemWeaver
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MemWeaverBackbone=GPT-4o-mini, Avg. Input Tokens=6722026.01 | 50.83 | 43.72 | 23.36 | |
| MemWeaverBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7342026.01 | 46.07 | 38.47 | 17.74 | |
| A-MemBackbone=GPT-4o-mini, Avg. Input Tokens=2,7312026.01 | 38.77 | 33.71 | 15.58 | |
| MemWeaverBackbone=Llama3.2-1B, Avg. Input Tokens=10002026.01 | 24.11 | 17.42 | 7.53 | |
| LoCoMoBackbone=GPT-4o-mini, Avg. Input Tokens=21,6252026.01 | 22.54 | 16.93 | 9.03 | |
| A-MemBackbone=Llama3.2-1B, Avg. Input Tokens=26472026.01 | 17.76 | 12.98 | 3.64 | |
| LoCoMoBackbone=Llama3.2-1B, Avg. Input Tokens=22,3122026.01 | 15.51 | 12.83 | 1.01 | |
| A-MemBackbone=Qwen2.5-1.5B, Avg. Input Tokens=2,5742026.01 | 14.54 | 12.45 | 1.73 | |
| ReadAgentBackbone=GPT-4o-mini, Avg. Input Tokens=6432026.01 | 12.6 | 8.87 | 0.95 | |
| MemWeaverBackbone=Llama3.2-3B, Avg. Input Tokens=10002026.01 | 11.37 | 13.16 | 8.67 | |
| A-MemBackbone=Llama3.2-3B, Avg. Input Tokens=25082026.01 | 8.62 | 6.76 | 4.41 | |
| LoCoMoBackbone=Qwen2.5-1.5B, Avg. Input Tokens=22,4242026.01 | 7.65 | 6.76 | 0.21 | |
| MemoryBankBackbone=GPT-4o-mini, Avg. Input Tokens=1,5132026.01 | 5.04 | 3.02 | 0.56 | |
| LoCoMoBackbone=Llama3.2-3B, Avg. Input Tokens=22,3122026.01 | 5 | 5.45 | 0.77 | |
| MemoryBankBackbone=Qwen2.5-1.5B, Avg. Input Tokens=1,5552026.01 | 3.43 | 2.85 | 0.2 | |
| ReadAgentBackbone=Llama3.2-3B, Avg. Input Tokens=4612026.01 | 3.01 | 3.01 | 3.01 | |
| ReadAgentBackbone=Qwen2.5-1.5B, Avg. Input Tokens=7522026.01 | 2.55 | 2.51 | 0 | |
| MemoryBankBackbone=Llama3.2-3B, Avg. Input Tokens=1,5532026.01 | 2.06 | 1.19 | 0.13 | |
| ReadAgentBackbone=Llama3.2-1B, Avg. Input Tokens=6652026.01 | 1.93 | 2.3 | 0 | |
| MemoryBankBackbone=Llama3.2-1B, Avg. Input Tokens=1,5532026.01 | 1.89 | 1.61 | 0.06 |