# Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

Source: [Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models](<https://www.sota2.com/research/paper/69cb555f0636e5bf4afca24b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-01T05:00:04.436Z

Authors: Baoheng Zhang, Jiahui Liu, Gui Zhao, Weizhou Zhang, Yixuan Ma, Jun Jiang, Yingxian Chen, Wilton W.T. Fok, Xiaojuan Qi, Hayden Kwok-Hay So

Published: 2026-03-29T07:46:32.000Z

## Abstract

Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-Choice on Extreme Illumination Triplet Instruct-Following Dataset](<https://www.sota2.com/research/sota/multi-choice-on-extreme-illumination-triplet-instruct-following-dataset.md>): Multi-Choice; Extreme Illumination Triplet Instruct-Following Dataset; Accuracy: 53.13.

- [Object-Counting on Extreme Illumination Triplet Instruct-Following Dataset](<https://www.sota2.com/research/sota/object-counting-on-extreme-illumination-triplet-instruct-following-dataset.md>): Object Counting; Extreme Illumination Triplet Instruct-Following Dataset; Accuracy: 74.66.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.27558>)

[PDF](<https://arxiv.org/pdf/2603.27558.pdf>)
