Deep Denoising for Multiview Depth Cameras

Quentin Bolsée¹, Leon Denis¹, Walid Darwish¹, Nastaran Nourbakhsh Kaashki¹ and Adrian Munteanu¹

¹Vrije Universiteit Brussel (VUB)

Abstract

A novel method for noise removal in multicamera depth sensing systems is proposed in this work. The method uses a combination of convolutional neural networks applied on each depth camera separately, followed by depth reprojection and joint processing of the resulting point clouds using a 3D neural network. Both depth and point cloud processing networks are designed to preserve the structure of the depth maps by appropriately correcting noise in the direction of the viewing rays. The proposed method accommodates any depth unit and noise intensity, thanks to adequate normalization in both the processing steps. The proposed approach is shown to outperform the state-of-the-art methods for both synthetic and real data captured with a multicamera setup and can reduce intercamera inconsistencies while preserving depth map structures.

Teaser figure — Overview of the system: the depth maps are first processed individually by a convolutional neural network, then jointly as a point cloud by a 3D neural network.

Method

The core contribution of this work is to process multiview depth data first as a set of 2D images, then as a joint 3D point cloud. In our workflow, noisy depth images are first processed indpenedently by a 2D convolution neural network trained on single camera depth noise removal. After this step, the point clouds from indpendent cameras will not match perfectly, as they were processed separately. To remedy this, the joint point cloud is then processed by a 3D neural network trained to solve this type of inconsistencies. The points are only allowed to be shifted along the rays defined by the optics of each depth camera, which allows the final joint point cloud to be reprojected to each depth camera without any loss of information.

Setup with two Kinect V2 sensors — Setup used for validation of the method, including a 3D printed object and two Kinect V2 sensors.

This method was tested on synthetic data as well as real data acquired from Kinect V2 sensors. The objects were 3D printed, scanned and measured against the reference 3D model. Our method systematically outperformed existing noise removal tools from the literature, and shows increased consistency across the multiple cameras.

Downloads

Source Code

Dataset

Publication

Citation

Bolsée, Q., Denis, L., Darwish, W., Kaashki, N. N., & Munteanu, A. (2022). Deep denoising for multiview depth cameras. IEEE Transactions on Instrumentation and Measurement, 71, 1-12.

@article{bolsee2022deep,
    title={Deep denoising for multiview depth cameras},
    author={Bols{\'e}e, Quentin and Denis, Leon and Darwish, Walid and Kaashki, Nastaran Nourbakhsh and Munteanu, Adrian},
    journal={IEEE Transactions on Instrumentation and Measurement},
    volume={71},
    pages={1--12},
    year={2022},
    publisher={IEEE}
}