# Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters

Source: [Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters](<https://www.sota2.com/research/paper/68fa68dc1e209131cea1ebb2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:47:52.778Z

Authors: Zhibo Zhang, Chen Zeng, Maulikkumar Dhameliya, Souma Chowdhury, Rahul Rai

Published: 2020-12-08T23:59:48.000Z

## Abstract

This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal data is processed through a thresholding and Kalman filter approach to detect and track the bounding box. Training and testing data are prepared by combining a set of original experiments conducted in a motion capture environment and publicly available UAV image data. The new pipeline compares favorably to existing methods and demonstrates promising tracking and localization capacity of sample experiments.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.04794>)

[PDF](<https://arxiv.org/pdf/2012.04794.pdf>)
