SAMURAI

Shape And Material from Unconstrained Real-world Arbitrary Image collections

Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan T. Barron, Hendrik P. A. Lensch, Varun Jampani

Presented By
Mark Boss
Relightable 3D assets from unposed image collections
Applications - AR
Applications - Material Editing
Applications - Games/Movies
Applications - Object Interaction
NeRF: Neural Radiance Fields
  • NeRF [1] is a method for photorealistic results in novel view synthesis
  • Main task is to learn where radiance is stored in a 3D neural field

[1] Mildenhall et al. - NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis - 2020

NeRD[1] and Neural-PIL[2]

[1] Boss et al. - NeRD: Neural Reflectance Decomposition from Image Collections - 2021

[2] Boss et al. - Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition - 2021

  • Extend NeRF to decompose radiance in reflectance and illumination
  • Enables relighting
COLMAP fails on objects in varying locations
Approach

Optimize camera parameters, neural reflectance field, and illumination

Coarse-to-Fine & Camera Multiplex
  • BARF-style Fourier Encoding Annealing
  • Gradual increase in resolution
  • Multiple camera estimates
Camera Multiplex
Image Posterior Scaling
  • Posterior scaling also applied on image level
  • Influence of badly aligned images or segmentation masks reduced

Influence of poorly aligned images is reduced

Varying Camera Distance
Results
Results
Comparison with BARF

Exemplary Inputs

BARF

SAMURAI

Results - Novel View Synthesis

Single Illumination
Method Pose Init PSNR ↑ Translation Error ↓ Rotation° Error ↓
BARF [1] Directions 14.96 34.64 0.86
GNeRF [2] Random 20.3 81.22 2.39
NeRS [3] Directions 12.84 32.77 0.77
SAMURAI Directions 21.08 33.95 0.71
NeRD GT 23.86 — —
Neural-PIL GT 23.95 — —

[1] Lin et al. - BARF: Bundle-adjusting neural radiance fields

[2] Meng et al. - GNeRF: GAN-based Neural Radiance Field without Posed Camera

[3] Zhang et al. - NeRS: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild

Results - Novel View Synthesis & Relighting

Dataset with poses available (NeRD datasets)
Method Pose Init PSNR ↑ Translation Error ↓ Rotation° Error ↓
BARF-A Directions 19.7 23.38 2.99
SAMURAI Directions 22.84 8.61 0.89
NeRD GT 26.88 — —
Neural-PIL GT 27.73 — —

New SAMURAI datasets (No poses recoverable)
Method Pose Init PSNR ↑
BARF-A Directions 16.9
SAMURAI Directions 23.46
Invisible

Thank you for listening

Project page available at:

markboss.me/publication/2022-samurai/