# Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision

Source: [Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision](<https://www.sota2.com/research/paper/68fa76021e209131cea6ae65>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T08:15:12.401Z

Authors: Jinnyeong Kim, Seung-Hwan Baek

Published: 2024-11-27T03:44:21.000Z

## Abstract

Integrating RGB and NIR stereo imaging provides complementary spectral information, potentially enhancing robotic 3D vision in challenging lighting conditions. However, existing datasets and imaging systems lack pixel-level alignment between RGB and NIR images, posing challenges for downstream vision tasks. In this paper, we introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot. The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points. Utilizing the mobility of the robot, we present a dataset containing continuous video frames under diverse lighting conditions. We then introduce two methods that utilize the pixel-aligned RGB-NIR images: an RGB-NIR image fusion method and a feature fusion method. The first approach enables existing RGB-pretrained vision models to directly utilize RGB-NIR information without fine-tuning. The second approach fine-tunes existing vision models to more effectively utilize RGB-NIR information. Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Depth Estimation on RGB-NIR](<https://www.sota2.com/research/sota/depth-estimation-on-rgb-nir.md>): Depth Estimation; RGB-NIR; RMSE: 7.567.

- [Object Detection on RGB-NIR](<https://www.sota2.com/research/sota/object-detection-on-rgb-nir.md>): Object Detection; RGB-NIR; mAP: 80.9.

- [Object Detection on RGB-NIR Dataset](<https://www.sota2.com/research/sota/object-detection-on-rgb-nir-dataset.md>): Object Detection; RGB-NIR Dataset; Detection mAP: 82.8.

- [Depth Estimation on RGB-NIR Stereo Dataset (test)](<https://www.sota2.com/research/sota/depth-estimation-on-rgb-nir-stereo-dataset-test.md>): Depth Estimation; RGB-NIR Stereo Dataset (test); Depth RMSE (m): 6.747.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.18025>)

[PDF](<https://arxiv.org/pdf/2411.18025.pdf>)