Lighting Estimation for Mobile AR (Part I)

Original Source Here

Lighting Estimation for Mobile AR (Part I)


In this article, I am going to briefly compare two recent research papers, Xihe and GLEAM, that provide real-time lighting estimation on mobile devices. Lighting estimation, briefly, refers to obtaining lighting information from environment scenes. For augmented reality, the scenes are physical environments (e.g., your living room) and therefore can have natural lighting condition changes, both temporally and spatially.

Dynamic scene lighting is a key reason that lighting estimation for physical environments is challenging. As lighting conditions can potentially change from one frame to the next, such rapid changes necessitate fast estimation (e.g., 30fps); additionally, lighting conditions are likely to vary from one corner of the room to the other corner (e.g., due to light source location), making it important to derive lighting estimation for positions of interests.

Existing commercial AR platforms including ARCore and ARKit have started to provide lighting estimate APIs, such as the ARLightEstimate. However, existing supports are still at an early stage, only providing ambient light information for an entire scene. In other words, existing commercial AR platforms including ARCore and ARKit still lack supports for accurate lighting estimation and thus often result in unrealistic rendering effects. Below are two diagrams that demonstrate the rendering effects using lighting information provided by ARKit vs. GLEAM/Xihe. Both GLEAM and Xihe produce more visually coherent virtual objects.

screenshot from the original GLEAM paper: AR scene rendering comparison.
screenshot from the original Xihe paper: AR scene rendering comparison.

GLEAM (MobiSys’2019)

In a nutshell, GLEAM uses physical light probes to capture environment lighting information which will then be provided to graphic renderers to illuminate virtual objects in the physical world.


What is a physical light probe? It is usually a reflective sphere that is placed at a specific physical location for sensing light. The diagram below shows an example light probe in the form of a chrome ball.

screenshot from the original GLEAM paper: an example physical light probe

What is a cubemap? A cubemap is one of the common representations of an environment map. An environment map helps us to model illumination by mapping incoming ray directions to ray intensity. The diagram below shows an example cubemap captured by GLEAM.

screenshot from the original GLEAM paper: an example cubemap.

GLEAM: Under the hood

GLEAM conceptually consists of the following two key steps.

Step 1: generating a radiance sample. To start, a GLEAM light probe (a physical light probe and an AR positioning marker) will be placed in the scene. Then, the AR user will capture images of the GLEAM light probe to generate radiance samples by associating each pixel with a reflected ray. The captured radiance samples will serve as the basis for constructing the environment map.

screenshot from the original GLEAM paper

Step 2: compositing a cubemap from multiple radiance samples. As the radiance samples are inherently sparse, e.g., limited by the number of light probes or FoVs, GLEAM uses a modified inverse distance weighting (IDW) interpolation algorithm and a nearest neighbor algorithm to assign values to each cubemap texel. The texel value assignment is a two-stage process.

screenshot from the original GLEAM paper

Specifically, each texel u(x) will be first populated by corresponding values from eligible radiance samples using the following equation. Here, wi(x) denotes the weight assigned to radiance sample i, and d(x, xi) describes the distance between the radiance sample i and the texel x. GLEAM uses a reliability-based weight function and L1 distance function. Theoretically, one can choose other weight and distance functions. For each texel, an eligible radiance sample has to satisfy a radius check. In the second stage, any empty texels will be filled using their neighboring texel values.

Wrap up

GLEAM presents an end-to-end framework for lighting estimation in mobile AR. As the paper stated, “GLEAM builds on a rich history of illumination estimation from the graphics community.” Indeed, the physical light probes-based approach is widely used outside mobile AR; but it is still pleasant to see this idea realized and demonstrated effectiveness for mobile AR.

If you are interested in working on physical probe-based lighting estimation, the authors have pointed out a number of limitations. For example, how to achieve better interpolation quality of cubemap from radiance samples while adhering to the real-time goal? What is the network impact on GLEAM’s performance, especially under multi-viewport configurations?


Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: