# Panoptic-Depth Forecasting

Source: [Panoptic-Depth Forecasting](<https://www.sota2.com/research/paper/68fa74d81e209131cea6417d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:19:27.756Z

Authors: Juana Valeria Hurtado, Riya Mohan, Abhinav Valada

Published: 2024-09-18T14:21:07.000Z

## Abstract

Forecasting the semantics and 3D structure of scenes is essential for robots to navigate and plan actions safely. Recent methods have explored semantic and panoptic scene forecasting; however, they do not consider the geometry of the scene. In this work, we propose the panoptic-depth forecasting task for jointly predicting the panoptic segmentation and depth maps of unobserved future frames, from monocular camera images. To facilitate this work, we extend the popular KITTI-360 and Cityscapes benchmarks by computing depth maps from LiDAR point clouds and leveraging sequential labeled data. We also introduce a suitable evaluation metric that quantifies both the panoptic quality and depth estimation accuracy of forecasts in a coherent manner. Furthermore, we present two baselines and propose the novel PDcast architecture that learns rich spatio-temporal representations by incorporating a transformer-based encoder, a forecasting module, and task-specific decoders to predict future panoptic-depth outputs. Extensive evaluations demonstrate the effectiveness of PDcast across two datasets and three forecasting tasks, consistently addressing the primary challenges. We make the code publicly available at https://pdcast.cs.uni-freiburg.de.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semantic Occupancy Forecasting on SemanticKITTI (val)](<https://www.sota2.com/research/sota/semantic-occupancy-forecasting-on-semantickitti-val.md>): Semantic Occupancy Forecasting; SemanticKITTI (val); mIoU (1s): 7.48.

- [3D Occupancy Prediction on Occ3D-nuScenes (val)](<https://www.sota2.com/research/sota/3d-occupancy-prediction-on-occ3d-nuscenes-val.md>): 3D Occupancy Prediction; Occ3D-nuScenes (val); mIoU: 15.33.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.12008>)

[PDF](<https://arxiv.org/pdf/2409.12008.pdf>)
