# Lumos : Empowering Multimodal LLMs with Scene Text Recognition

Source: [Lumos : Empowering Multimodal LLMs with Scene Text Recognition](<https://www.sota2.com/research/paper/68fa73661e209131cea5c348>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:48:16.237Z

Authors: Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar

Published: 2024-02-12T19:27:26.000Z

## Abstract

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-Video Generation on VBench (Quality Metrics)](<https://www.sota2.com/research/sota/text-to-video-generation-on-vbench-quality-metrics.md>): Text-to-Video Generation; VBench; Quality Score: 79.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.08017>)

[PDF](<https://arxiv.org/pdf/2402.08017.pdf>)
