Rendering equation [1]

$$ \definecolor{out}{RGB}{219,135,217} \definecolor{emit}{RGB}{125,194,103} \definecolor{int}{RGB}{127,151,236} \definecolor{in}{RGB}{225,145,83} \definecolor{brdf}{RGB}{0,202,207} \definecolor{ndl}{RGB}{235,120,152} \definecolor{point}{RGB}{232,0,19} \color{out}L_{o}(\color{point}{\mathbf x}\color{out},\,\omega_{o})\color{black}\,= \fragment{1}{\,\color{emit}L_{e}({\mathbf x},\,\omega_{o})} \fragment{2}{\color{black} + \\ \color{int}\int_{\Omega }} \fragment{4}{\color{brdf}f_{r}({\mathbf x},\,\omega_{i},\,\omega_{o})} \fragment{3}{\color{in}L_{i}({\mathbf x},\,\omega_{i})}\, \fragment{5}{\color{ndl}(\omega_{i}\,\cdot\,{\mathbf n})}\, \fragment{2}{\color{int}\operatorname d\omega_{i}}$$

Simplification: No self-emittance

$$ \definecolor{out}{RGB}{219,135,217} \definecolor{emit}{RGB}{125,194,103} \definecolor{int}{RGB}{127,151,236} \definecolor{in}{RGB}{225,145,83} \definecolor{brdf}{RGB}{0,202,207} \definecolor{ndl}{RGB}{235,120,152} \definecolor{point}{RGB}{232,0,19} \color{out}L_{o}(\color{point}{\mathbf x}\color{out},\,\omega_{o})\color{black}\,=\,\color{int}\int_{\Omega} \color{brdf}f_{r}({\mathbf x},\,\omega_{i},\,\omega_{o}) \color{in}L_{i}({\mathbf x},\,\omega_{i})\, \color{ndl}(\omega_{i}\,\cdot\,{\mathbf n})\, \color{int}\operatorname d\omega_{i}$$

Radiance, reflectance and irradiance

$$\underbrace{L_{o}({\mathbf x},\,\omega_{o})}_{\text{Radiance (Outgoing)}}\,=\,\int_{\Omega}\underbrace{f_{r}({\mathbf x},\,\omega_{i},\,\omega_{o})}_{\text{Reflectance}} \\ \underbrace{L_{i}({\mathbf x},\,\omega_{i})\, (\omega_{i}\,\cdot\,{\mathbf n})\, \operatorname d\omega_{i}}_{\text{Irradiance}}$$

Plenoptic function

• Main use-case: novel view synthesis
• Learns the radiance
• No relighting possible

Intrinsic image

• Splits an image into layers:
  • Shading (Irradiance)
  • Diffuse albedo
  • Potential: Specular shading + albedo
• Partial decomposition of the rendering equation
• Albedos are accurate up to a shift and scale

Full decomposition

• Decomposes the rendering equation
• Reflectance modelled as BRDF
  • Often analytical
• Illumination as incoming radiance
• Often differentiable rendering is used for optimization

• Assume dataset of 4 images
• Images are varying illumination

Potential solution

• Train GLO (Generative Latent Optimization) to express radiance per illumination
• Interpolate between illuminations

• Only interpolate between seen illuminations
• No issue if dataset is vast and contains most illuminations
• However, it is hard to get all possible illumination edge cases
  
  

• Temporarily also known as: Coordinate-based MLPs
• Early works encode points $p \in \mathbb{R}^3$ in network as
  • Occupancy [1, 2]: $f(p) \rightarrow o; o \in {0, 1}$
  • Signed Distance Field [3]: $f(p) \rightarrow d; d \in \mathbb{R}$
• Only encode existing point clouds or models in the neural field

• Neural volumetric rendering first introduced by Lombardi et al. [1]
  • Accumulate opacity along a ray based on volume rendering
    
    

• Learn radiance (view-dependent RGB color)
• Goal is often novel view synthesis
• Often neural volume rendering is used [1, 2, 3]
• Can be extended with GLO embeddings to multiple illuminations [3]
  
  

• Decompose the rendering equation into geometry, illumination & reflectance
• Neural volume rendering (NeRF-style) used [1, 2, 3, 4]
• Can be also used to decompose a trained NeRF model [3]
• Specialized methods exist which directly optimize assets [4]
• Related works focus on single illumination per object
  
  

• Spherical Gaussian (SG) have convenient properties
• Close-form solution exists for integration 2 SGs
• Product of 2 SGs is a SG

• Extracting textured meshes allows multiple use cases

• Learn a smooth low-dimensional manifold to represent BRDF and lighting
• Constraints BRDF and light to plausible values
• Auto-encoder learning with interpolated latent space
• Smooth manifold losses include:
  • Adversarial
  • Gradient regularization during manifold interpolation

• Pre-compute light integrals for fast rendering

• Light pre-integration is still expensive
• We need to do the pre-integration on the fly
• Neural-PIL that converts light pre-integration into a simple network query
• Architecture based on pi-GAN [1]

Light Pre-integration Equation:

$$\tilde{L}_i(\omega_r, c_r) = \int_\Omega D(c_r, \omega_i, \omega_r)L_i(x, \omega_i)d\omega_i$$

• BARF-style Fourier Encoding Annealing
• Gradual increase in resolution
• Multiple camera estimates

• Posterior scaling also applied on image level
• Influence of badly aligned images or segmentation masks reduced

Influence of poorly aligned images is reduced

• Solving the inverse rendering problem is highly ill-posed

• Create 3D assets from unconstrained input data

• Currently only static scenes
• Even scenes without humans move due to wind (Trees)
• NeRFies [1] or HyperNeRF [2] bend the ray based on the time step
• Both NeRFies and HyperNeRF only output the radiance

• Decompose rooms or large buildings (churches)
• Illumination is not global and inter-reflections play a large role
• Modeling dynamic movement of foliage is important
• BlockNeRF only models radiance