Two-shot Spatially-varying BRDF and Shape Estimation
In CVPR
Capturing the shape and spatially-varying appearance (SVBRDF) of an object from images is a challenging task that has applications in both computer vision and graphics. Traditional optimization-based approaches often need a large number of images taken from multiple views in a controlled environment. Newer deep learning-based approaches require only a few input images, but the reconstruction quality is not on par with optimization techniques. We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. The previous predictions guide each estimation, and a joint refinement network later refines both SVBRDF and shape. We follow a practical mobile image capture setting and use unaligned two-shot flash and no-flash images as input. Both our two-shot image capture and network inference can run on mobile hardware. We also create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Extensive experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images. Comparisons with recent approaches demonstrate the superior performance of the proposed approach.
Single Image BRDF Parameter Estimation with a Conditional Adversarial Network
Creating plausible surfaces is an essential component in achieving a high degree of realism in rendering. To relieve artists, who create these surfaces in a time-consuming, manual process, automated retrieval of the spatially-varying Bidirectional Reflectance Distribution Function (SVBRDF) from a single mobile phone image is desirable. By leveraging a deep neural network, this casual capturing method can be achieved. The trained network can estimate per pixel normal, base color, metallic and roughness parameters from the Disney BRDF. The input image is taken with a mobile phone lit by the camera flash. The network is trained to compensate for environment lighting and thus learned to reduce artifacts introduced by other light sources. These losses contain a multi-scale discriminator with an additional perceptual loss, a rendering loss using a differentiable renderer, and a parameter loss. Besides the local precision, this loss formulation generates material texture maps which are globally more consistent. The network is set up as a generator network trained in an adversarial fashion to ensure that only plausible maps are produced. The estimated parameters not only reproduce the material faithfully in rendering but capture the style of hand-authored materials due to the more global loss terms compared to previous works without requiring additional post-processing. Both the resolution and the quality is improved.
Deep Dual Loss BRDF Parameter Estimation
In MAM
Surface parameter estimation is an essential field in computer games and movies. An exact representation of a real-world surface allows for a higher degree of realism. Capturing or artistically creating these materials is a time-consuming process. We propose a method which utilizes an encoder-decoder Convolutional Neural Network (CNN) to extract parameters for the Bidirectional Reflectance Distribution Function (BRDF) automatically from a sparse sample set. This is done by implementing a differentiable renderer, which allows for a loss backpropagation of rendered images. This photometric loss is essential because defining a numerical BRDF distance metric is difficult. A second loss is added, which compares the parameters maps directly. Therefore, the statistical properties of the BRDF model are learned, which reduces artifacts in the predicted parameters. This dual loss principal improves the result of the network significantly. Opposed to previous means this method retrieves information of the whole surface as spatially varying BRDF (SVBRDF) parameters with a sufficiently high resolution for intended real-world usage. The capture process for materials only requires five known light positions with a fixed camera position. This reduces the scanning time drastically, and a material sample can be obtained in seconds with an automated system.