Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning
About
Deep Q-Networks (DQN) can suffer from overestimation bias because bootstrapped targets use a maximisation operation over noisy value estimates. Ensemble-based methods and multi-step methods have each been used to improve the stability and sample efficiency of value-based reinforcement learning, but their interaction remains less well understood. This paper introduces Ensemble Elastic DQN (EEDQN), a value-based reinforcement learning algorithm that combines adaptive elastic multi-step returns with ensemble-based target aggregation. EEDQN replaces the clustering-based state similarity test used in earlier Elastic Step DQN with a lightweight Q-value difference rule, making adaptive return construction simpler to apply in discrete control settings. The method then applies horizon dependent ensemble aggregation, one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. This design aims to reduce extreme optimistic bootstrap estimates without making every update uniformly conservative. We evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN. EEDQN achieves the highest final return in four of the five environments and remains competitive with conservative ensemble baselines. An aggregation rule ablation shows that the best degree of conservatism is environment dependent, suggesting that adaptive return length and ensemble aggregation interact in non-trivial ways.