Interactive Database Interaction on BIRD-Interact Phase 2
34.3Success Rate (WRITE)MERIT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| MERITMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 34.3 | 17.22 | 22.17 | |
| MemRLMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 30.46 | 14.08 | 18.83 | |
| BM25Memory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 28.74 | 13.85 | 18.17 | |
| Cosine SimilarityMemory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 28.74 | 13.62 | 18 | |
| MementoMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 28.74 | 13.38 | 17.83 | |
| DemoMemory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 28.16 | 11.97 | 16.67 | |
| TopicKMemory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 25.86 | 11.97 | 16 | |
| No MemoryMemory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 20.11 | 10.33 | 13.17 | |
| Vanilla GPT-5Memory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 12.07 | 3.05 | 5.67 |