# Unifying Foundation Models with Quadrotor Control for Visual Tracking Beyond Object Categories

Source: [Unifying Foundation Models with Quadrotor Control for Visual Tracking Beyond Object Categories](<https://www.sota2.com/research/paper/68fa72d71e209131cea58d7b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:35:49.064Z

Authors: Alessandro Saviolo, Pratyaksh Rao, Vivek Radhakrishnan, Jiuhong Xiao, Giuseppe Loianno

Published: 2023-10-07T11:47:58.000Z

## Abstract

Visual control enables quadrotors to adaptively navigate using real-time sensory data, bridging perception with action. Yet, challenges persist, including generalization across scenarios, maintaining reliability, and ensuring real-time responsiveness. This paper introduces a perception framework grounded in foundation models for universal object detection and tracking, moving beyond specific training categories. Integral to our approach is a multi-layered tracker integrated with the foundation detector, ensuring continuous target visibility, even when faced with motion blur, abrupt light shifts, and occlusions. Complementing this, we introduce a model-free controller tailored for resilient quadrotor visual tracking. Our system operates efficiently on limited hardware, relying solely on an onboard camera and an inertial measurement unit. Through extensive validation in diverse challenging indoor and outdoor environments, we demonstrate our system's effectiveness and adaptability. In conclusion, our research represents a step forward in quadrotor visual tracking, moving from task-specific methods to more versatile and adaptable operations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Vision-based Aerial Tracking on FalconGym Gate 2.0](<https://www.sota2.com/research/sota/vision-based-aerial-tracking-on-falcongym-gate-2-0.md>): Vision-based Aerial Tracking; FalconGym Gate 2.0; Success Rate: 100.

- [Vision-based Aerial Tracking on Real World Gate](<https://www.sota2.com/research/sota/vision-based-aerial-tracking-on-real-world-gate.md>): Vision-based Aerial Tracking; Real World Gate; Success Rate: 100.

- [Vision-based Aerial Tracking on FalconGym Quadrotor 2.0](<https://www.sota2.com/research/sota/vision-based-aerial-tracking-on-falcongym-quadrotor-2-0.md>): Vision-based Aerial Tracking; FalconGym Quadrotor 2.0; Success Rate: 60.

- [Vision-based Aerial Tracking on FalconGym F1-tenth 2.0](<https://www.sota2.com/research/sota/vision-based-aerial-tracking-on-falcongym-f1-tenth-2-0.md>): Vision-based Aerial Tracking; FalconGym F1-tenth 2.0; Success Rate: 80.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2310.04781>)

[PDF](<https://arxiv.org/pdf/2310.04781.pdf>)
