ROODI: Reconstructing Occluded Objects with Denoising Inpainters


Yeonjin Chang1, Erqun Dong2, Seunghyeon Seo1, Nojun Kwak1, Kwang Moo Yi2
1Seoul National University, 2University of British Columbia

Descriptive Text

Abstract

While the quality of novel-view images has improved dramatically with 3D Gaussian Splatting, extracting specific objects from scenes remains challenging. Isolating individual 3D Gaussian primitives for each object and handling occlusions in scenes remain far from being solved. We propose a novel object extraction method based on two key principles: (1) being object-centric by pruning irrelevant primitives; and (2) leveraging generative inpainting to compensate for missing observations caused by occlusions. For pruning, we analyze the local structure of primitives using K-nearest neighbors, and retain only relevant ones. For inpainting, we employ an off-the-shelf diffusion-based inpainter combined with occlusion reasoning, utilizing the 3D representation of the entire scene. Our findings highlight the crucial synergy between pruning and inpainting, both of which significantly enhance extraction performance. We evaluate our method on a standard real-world dataset and introduce a synthetic dataset for quantitative analysis. Our approach outperforms the state-of-the-art, demonstrating its effectiveness in object extraction from complex scenes.

Object extraction on LERF dataset

figurines


Object extraction on MultiObjectBlender dataset

bedroom
kitchen
livingroom
picnic