Interactive Database Interaction on BIRD-Interact Phase 1
54.6Write Success RateDemo
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DemoMemory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 54.6 | 18.31 | 28.83 | |
| MERITMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 53.49 | 24.64 | 33.01 | |
| BM25Memory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 52.87 | 19.48 | 29.17 | |
| Cosine SimilarityMemory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 51.72 | 19.48 | 28.83 | |
| MemRLMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 51.15 | 19.95 | 29 | |
| MementoMemory Strategy=Dynamic Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 50.57 | 19.72 | 28.67 | |
| TopicKMemory Strategy=Static Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 48.85 | 18.31 | 27.17 | |
| No MemoryMemory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 43.68 | 15.02 | 23.33 | |
| Vanilla GPT-5Memory Strategy=No Memory, Backbone=GPT-5, User Simulator=GPT-4o2026.05 | 24.71 | 8.69 | 13.33 |