# HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities Source: [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:15:11.386Z Authors: Shristi Das Biswas, Adarsh Kosta, Chamika Liyanagedera, Marco Apolinario, Kaushik Roy Published: 2022-11-19T17:09:50.000Z ## Abstract Event cameras detect changes in per-pixel intensity to generate asynchronous \`event streams'. They offer great potential for accurate semantic map retrieval in real-time autonomous systems owing to their much higher temporal resolution and high dynamic range (HDR) compared to conventional cameras. However, existing implementations for event-based segmentation suffer from sub-optimal performance since these temporally dense events only measure the varying component of a visual signal, limiting their ability to encode dense spatial context compared to frames. To address this issue, we propose a hybrid end-to-end learning framework HALSIE, utilizing three key concepts to reduce inference cost by up to $20\\times$ versus prior art while retaining similar performance: First, a simple and efficient cross-domain learning scheme to extract complementary spatio-temporal embeddings from both frames and events. Second, a specially designed dual-encoder scheme with Spiking Neural Network (SNN) and Artificial Neural Network (ANN) branches to minimize latency while retaining cross-domain feature aggregation. Third, a multi-scale cue mixer to model rich representations of the fused embeddings. These qualities of HALSIE allow for a very lightweight architecture achieving state-of-the-art segmentation performance on DDD-17, MVSEC, and DSEC-Semantic datasets with up to $33\\times$ higher parameter efficiency and favorable inference cost (17.9mJ per cycle). Our ablation study also brings new insights into effective design choices that can prove beneficial for research across other vision tasks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Semantic Segmentation on MVSEC (outdoor day2)](): Semantic segmentation; MVSEC (outdoor day2); mIoU: 66.31. - [Semantic Segmentation on DSEC-Semantic (test)](): Semantic segmentation; DSEC-Semantic (test); Accuracy: 0.8901. - [Event-based Semantic Segmentation on DDD17 (test)](): Event-based Semantic Segmentation; DDD17 (test); mIoU (General): 60.66. - [Event-based Semantic Segmentation on DSEC-Semantic (test)](): Event-based Semantic Segmentation; DSEC-Semantic (test); Accuracy: 89.01. - [Semantic Segmentation on DSEC](): Semantic segmentation; DSEC; mIoU: 52.43. - [Semantic Segmentation on DSEC-Semantic](): Semantic segmentation; DSEC-Semantic; mIoU: 52.43. - [Semantic Segmentation on DDD17 (test)](): Semantic segmentation; DDD17 (test); mIoU: 60.66. - [Semantic Segmentation on DSEC (test)](): Semantic segmentation; DSEC (test); mIoU: 52.43. - [Semantic Segmentation on DDD17](): Semantic segmentation; DDD17; mIoU: 60.66. ## Original sources and implementation [Original paper]() [PDF]()