# Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

Source: [Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion](<https://www.sota2.com/research/paper/6a210684c10484bac23f00e1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-04T05:00:05.280Z

Authors: Oskar Natan, Jun Miura

Published: 2026-06-02T00:35:42.000Z

## Abstract

We present a novel compact deep multi-task learning model to handle various autonomous driving perception tasks in one forward pass. The model performs multiple views of semantic segmentation, depth estimation, light detection and ranging (LiDAR) segmentation, and bird's eye view projection simultaneously without being supported by other models. We also provide an adaptive loss weighting algorithm to tackle the imbalanced learning issue that occurred due to plenty of given tasks. Through data pre-processing and intermediate sensor fusion techniques, the model can process and combine multiple input modalities retrieved from RGB cameras, dynamic vision sensors (DVS), and LiDAR placed at several positions on the ego vehicle. Therefore, a better understanding of a dynamically changing environment can be achieved. Based on the ablation study, the model variant trained with our proposed method achieves a better performance. Furthermore, a comparative study is also conducted to clarify its performance and effectiveness against the combination of some recent models. As a result, our model maintains better performance even with much fewer parameters. Hence, the model can inference faster with less GPU memory utilization. Moreover, the result tends to be consistent in 3 different CARLA simulation datasets and 1 real-world nuScenes-lidarseg dataset. To support future research, we share codes and other files publicly at https://github.com/oskarnatan/compact-perception.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-task Autonomous Driving Perception on CARLA Simulation Set B (test)](<https://www.sota2.com/research/sota/multi-task-autonomous-driving-perception-on-carla-simulation-set-b-test.md>): Multi-task Autonomous Driving Perception; CARLA Simulation Set B (test); IoU (BEVP): 63.

- [Multi-task Autonomous Driving Perception on nuScenes-lidarseg (test)](<https://www.sota2.com/research/sota/multi-task-autonomous-driving-perception-on-nuscenes-lidarseg-test.md>): Multi-task Autonomous Driving Perception; nuScenes-lidarseg (test); IoU (BEVP): 83.3.

- [Multi-task Autonomous Driving Perception on CARLA Simulation Set A (test)](<https://www.sota2.com/research/sota/multi-task-autonomous-driving-perception-on-carla-simulation-set-a-test.md>): Multi-task Autonomous Driving Perception; CARLA Simulation Set A (test); IoU (BEV Projection): 61.3.

- [Multi-task Autonomous Driving Perception on CARLA Simulation Set C (test)](<https://www.sota2.com/research/sota/multi-task-autonomous-driving-perception-on-carla-simulation-set-c-test.md>): Multi-task Autonomous Driving Perception; CARLA Simulation Set C (test); mIoU (BEV): 63.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.02979>)

[PDF](<https://arxiv.org/pdf/2606.02979.pdf>)
