# HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

Source: [HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning](<https://www.sota2.com/research/paper/6a3a1354206542251d7c7323>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T05:00:13.022Z

Authors: Awais Rauf, Ahmed Hasssan, Greg Slabaugh

Published: 2026-06-19T20:43:49.000Z

## Abstract

Understanding long videos requires fine-grained perception and multi-step, higher-order reasoning over complex, long-range spatio-temporal dynamics. Vision-language models (VLMs) encode video frames into visual tokens and attempt to perform both perception and multi-step planning latently, within a single forward pass. This coupled formulation, however, is bottlenecked by the LLM's limited capacity to discover and execute multi-step strategies in its latent representations. To address this bottleneck, we propose Hierarchical Programmatic Probing (HPP), a framework that decouples semantic perception from higher-order temporal reasoning by reformulating long video understanding as iterative, programmatic exploration of a hierarchically segmented video. Specifically, a coding-capable LLM plans and executes a multi-step strategy in an interactive coding environment, probing the video for information and invoking a VLM for localized perception on demand. To make probing tractable over long videos, we introduce three components: information-density-aware hierarchical segmentation, late-interaction semantic retrieval, and structured probing functions for coarse-to-fine temporal localization. We validate HPP on LongVideoBench, which requires both fine-grained perception and long-range relational reasoning, and show that decoupling the two via iterative programmatic probing yields substantial gains. Further results on EgoSchema, VideoMME, and MLVU demonstrate the effectiveness of our approach across diverse long-video benchmarks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Accuracy on LongVideoBench (Long-video understanding)](<https://www.sota2.com/research/sota/accuracy-on-longvideobench-long-video-understanding.md>): Long Video Understanding; LongVideoBench average; Accuracy: 66.4.

- [Long-video understanding on EgoSchema (full)](<https://www.sota2.com/research/sota/long-video-understanding-on-egoschema-full.md>): Long Video Understanding; EgoSchema (Full); Accuracy: 70.4.

- [Long-video understanding on VideoMME (average)](<https://www.sota2.com/research/sota/long-video-understanding-on-videomme-average.md>): Long Video Understanding; VideoMME (average); Accuracy: 71.8.

- [Long Video Understanding on MLVU](<https://www.sota2.com/research/sota/long-video-understanding-on-mlvu.md>): Long Video Understanding; MLVU; Accuracy: 69.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.21734>)

[PDF](<https://arxiv.org/pdf/2606.21734.pdf>)
