NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects
About
Dynamic Neural Radiance Field (NeRF) is a powerful algorithm capable of rendering photo-realistic novel view images from a monocular RGB video of a dynamic scene. Although it warps moving points across frames from the observation spaces to a common canonical space for rendering, dynamic NeRF does not model the change of the reflected color during the warping. As a result, this approach often fails drastically on challenging specular objects in motion. We address this limitation by reformulating the neural radiance field function to be conditioned on surface position and orientation in the observation space. This allows the specular surface at different poses to keep the different reflected colors when mapped to the common canonical space. Additionally, we add the mask of moving objects to guide the deformation field. As the specular surface changes color during motion, the mask mitigates the problem of failure to find temporal correspondences with only RGB supervision. We evaluate our model based on the novel view synthesis quality with a self-collected dataset of different moving specular objects in realistic environments. The experimental results demonstrate that our method significantly improves the reconstruction quality of moving specular objects from monocular RGB videos compared to the existing NeRF models. Our code and data are available at the project website https://github.com/JokerYan/NeRF-DS.
Key insights
What problem does NeRF-DS solve?
Dynamic NeRFs warp moving points to a shared canonical space but ignore changing specular reflections, causing failures on shiny moving objects. NeRF-DS conditions color on observation-space position and surface normal, plus adds moving-object masks to guide deformation.
How does it differ from prior work?
Unlike HyperNeRF, Nerfies, and Ref-NeRF, NeRF-DS handles both motion and specularity. On an 8-scene self-collected dataset at 480x270, it improves mean LPIPS to 0.135 versus HyperNeRF's 0.192 and PSNR to 23.4 versus 22.7.
What is it useful for in practice?
NeRF-DS improves novel-view synthesis of moving specular objects from monocular RGB video, useful for reconstructing metallic or plastic surfaces. The authors release code and a dynamic specular dataset but document no production deployment.
What are the stated limitations?
NeRF-DS relies on accurate surface normal predictions, but reflected texture can mislead predicted geometry, making normals follow reflections rather than true surface shape. The authors leave surface priors and constrained deformation models to future work.