Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning

About

Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.

Yingxiao Huo, Satya Prakash Dash, Radu Stoican, Samuel Kaski, Mingfei Sun · 2026

Related benchmarks

Task	Dataset	Result
Reinforcement Learning	Walker	Average Returns204.1	38
Reinforcement Learning	Humanoid	Zero-Shot Reward4625	32
Reinforcement Learning	cartpole	Average Reward973.4	29
Reinforcement Learning	Pendulum	Avg Episode Reward-2226	26
Reinforcement Learning	Swimmer	Average Returns28.5	24
Reinforcement Learning	Halfcheetah	Average Return2936	22
Reinforcement Learning	Pusher	Average Returns-408.2	16
Reinforcement Learning	Hopper	Avg Episode Reward331.2	15
Reinforcement Learning	Acrobot	Average Returns-94.1	10