# Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

Source: [Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline](<https://www.sota2.com/research/paper/69aa5f9b2d603e6de3e41a87>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-09T06:00:01.064Z

Authors: Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, Zhenquan Li, Xinyi Mao, Baoqi Pei, Shihao Wang, Zhiqi Li, Karan Sapra, Fuxiao Liu, Yin-Dong Zheng, Yifei Huang, Limin Wang, Zhiding Yu, Andrew Tao, Guilin Liu, Tong Lu

Published: 2026-03-05T18:52:12.000Z

## Abstract

While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multimodal Question Answering on MM-Lifelong (val@month)](<https://www.sota2.com/research/sota/multimodal-question-answering-on-mm-lifelong-val-at-month.md>): Multimodal Question Answering; MM-Lifelong (val@month); Accuracy: 18.62.

- [Multimodal Question Answering on MM-Lifelong week (test)](<https://www.sota2.com/research/sota/multimodal-question-answering-on-mm-lifelong-week-test.md>): Multimodal Question Answering; MM-Lifelong week (test); Accuracy: 18.82.

- [Multimodal Question Answering on MM-Lifelong @day (test)](<https://www.sota2.com/research/sota/multimodal-question-answering-on-mm-lifelong-at-day-test.md>): Multimodal Question Answering; MM-Lifelong @day (test); Accuracy: 16.75.

- [Long-horizon video understanding on MM-Lifelong (train@month)](<https://www.sota2.com/research/sota/long-horizon-video-understanding-on-mm-lifelong-train-at-month.md>): Long-horizon video understanding; MM-Lifelong (train@month); Accuracy: 17.62.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.05484>)

[PDF](<https://arxiv.org/pdf/2603.05484.pdf>)

[Implementation](<https://github.com/cg1177/Recursive-Multimodal-Agent>)