# Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning

Source: [Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning](<https://www.sota2.com/research/paper/693ba16b6cfd488761a55090>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-10T05:00:04.898Z

Authors: Chihyeon Song, Jaewoo Lee, Jinkyoo Park

Published: 2025-12-11T10:30:04.000Z

## Abstract

Offline-to-Online Reinforcement Learning (O2O RL) faces a critical dilemma in balancing the use of a fixed offline dataset with newly collected online experiences. Standard methods, often relying on a fixed data-mixing ratio, struggle to manage the trade-off between early learning stability and asymptotic performance. To overcome this, we introduce the Adaptive Replay Buffer (ARB), a novel approach that dynamically prioritizes data sampling based on a lightweight metric we call 'on-policyness'. Unlike prior methods that rely on complex learning procedures or fixed ratios, ARB is designed to be learning-free and simple to implement, seamlessly integrating into existing O2O RL algorithms. It assesses how closely collected trajectories align with the current policy's behavior and assigns a proportional sampling weight to each transition within that trajectory. This strategy effectively leverages offline data for initial stability while progressively focusing learning on the most relevant, high-rewarding online experiences. Our extensive experiments on D4RL benchmarks demonstrate that ARB consistently mitigates early performance degradation and significantly improves the final performance of various O2O RL algorithms, highlighting the importance of an adaptive, behavior-aware replay buffer design. Our code is publicly available at https://github.com/song970407/ARB.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Offline-to-Online Reinforcement Learning on D4RL Locomotion (random)](<https://www.sota2.com/research/sota/offline-to-online-reinforcement-learning-on-d4rl-locomotion-random.md>): Offline-to-Online Reinforcement Learning; D4RL Locomotion random; Avg Normalized Return: 53.1.

- [Offline-to-Online Reinforcement Learning on D4RL Locomotion medium-replay](<https://www.sota2.com/research/sota/offline-to-online-reinforcement-learning-on-d4rl-locomotion-medium-replay.md>): Offline-to-Online Reinforcement Learning; D4RL Locomotion medium-replay; Avg Normalized Return: 90.8.

- [Offline-to-Online Reinforcement Learning on D4RL Locomotion medium](<https://www.sota2.com/research/sota/offline-to-online-reinforcement-learning-on-d4rl-locomotion-medium.md>): Offline-to-Online Reinforcement Learning; D4RL Locomotion medium; Average Normalized Return: 98.3.

- [Offline-to-Online Reinforcement Learning on D4RL Locomotion medium-expert](<https://www.sota2.com/research/sota/offline-to-online-reinforcement-learning-on-d4rl-locomotion-medium-expert.md>): Offline-to-Online Reinforcement Learning; D4RL Locomotion medium-expert; Average Normalized Return: 107.9.

- [Offline-to-Online Reinforcement Learning on D4RL Antmaze](<https://www.sota2.com/research/sota/offline-to-online-reinforcement-learning-on-d4rl-antmaze.md>): Offline-to-Online Reinforcement Learning; D4RL AntMaze; Avg Normalized Return: 91.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.10510>)

[PDF](<https://arxiv.org/pdf/2512.10510.pdf>)
