# Hierarchical Zero-Order Optimization for Deep Neural Networks Source: [Hierarchical Zero-Order Optimization for Deep Neural Networks]() Collection index: [llms.txt]() Data updated: 2026-02-13T05:00:04.184Z Authors: Sansheng Cao, Zhengyu Ma, Yonghong Tian Published: 2026-02-11T07:56:07.000Z ## Abstract Zeroth-order (ZO) optimization has long been favored for its biological plausibility and its capacity to handle non-differentiable objectives, yet its computational complexity has historically limited its application in deep neural networks. Challenging the conventional paradigm that gradients propagate layer-by-layer, we propose Hierarchical Zeroth-Order (HZO) optimization, a novel divide-and-conquer strategy that decomposes the depth dimension of the network. We prove that HZO reduces the query complexity from $O(ML^2)$ to $O(ML \\log L)$ for a network of width $M$ and depth $L$, representing a significant leap over existing ZO methodologies. Furthermore, we provide a detailed error analysis showing that HZO maintains numerical stability by operating near the unitary limit ($L\_{lip} \\approx 1$). Extensive evaluations on CIFAR-10 and ImageNet demonstrate that HZO achieves competitive accuracy compared to backpropagation. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on CIFAR-10 (test)](): Image Classification; CIFAR-10 (test); Accuracy: 74.2. ## Original sources and implementation [Original paper]() [PDF]()