Non-line-of-sight reconstruction with signal-object collaborative regularization

Non-line-of-sight (NLOS) imaging aims at recovering obscured objects from multiple scattered light. Our recent work provides a signal-object collaborative regularization (SOCR) framework for high-quality and noise-robust NLOS reconstructions.

Like Comment
Read the paper

Non-line-of-sight technology enables us to detect objects out of the direct line of sight. In real applications, lasers or other light sources are used to illuminate a visible wall, the scattered light from which reaches the hidden object and is scattered back again. The photons collected by detectors are used to recover the location, shape, albedo, and normal of the target. Potential applications of this technology include autonomous driving, survivor rescuing, and remote sensing.

The measured data is inevitably corrupted with ambient noise. The signal-noise ratio is usually low due to multiple reflections of the photons before they are detected. Reconstructions obtained with existing methods are usually blurry with heavy background noise, especially in the case of short exposure time. A broad class of approaches introduces regularization of the hidden object to enhance the reconstruction quality. Commonly used priors include L1, L2 and total variation. In practice, these priors are not enough to obtain clear reconstructions due to heavy measurement noise and the complex nature of real-world objects.

Reconstructed albedo and surface normal of different scenarios. (a) Synthetic, confocal, from Zaragoza dataset ( (b) Measured, confocal, indoor, from Stanford dataset ( (c) Measured, confocal, indoor, from Stanford dataset ( (d) Measured, confocal, outdoor, from Stanford dataset  ( (e) Measured, non-confocal, indoor, from the dataset provided by the phasor field method (

In our recent work, we propose a signal-object collaborative regularization (SOCR) framework for NLOS reconstructions. In this framework, signal and reconstructed target are optimized simultaneously. The proposed collaborative regularization incorporates sparseness and non-local self-similarity of the hidden object as well as smoothness of the signal. The novelty of our work includes

  1. The albedo and surface normal can be reconstructed under both confocal and non-confocal settings.
  2. An estimation of the oracle signal corresponding to the reconstructed target is obtained, which gives insights to noise estimation of the raw measurements.
  3. Two dictionaries are used to capture the local spatial structures and non-local correlations of the hidden targets, which can be beneficial for further tasks, such as recognition and classification.
  4. The proposed framework is robust in practical applications. Even in cases where the measurement noise is high, our algorithm still reconstructs the hidden objects faithfully with clear local structures and sharp boundaries. 

The designed collaborative regularization is a combination of three priors of the signal and the object. It is necessary to introduce them all for high-quality reconstructions of the unseen target. The figure below shows reconstruction results of the instance of the dragon from the Stanford dataset in extremely low exposure time with different combinations of the priors.

Reconstruction results of the dragon with different regularizations (confocal, 15 s). The measured signal is provided in the Stanford dataset ( The original signal is downsampled to 64×64 points. The measurements contain heavy noise due to an extremely short exposure time. The proposed joint signal object prior is a combination of three priors, namely (I) the sparseness prior of the target, (II) the non-local self-similarity prior of the target, and (III) the smoothness prior of the signals.

The proposed SOCR reconstruction algorithm is solved using the alternative iteration method with two stages. In the initializing stage, a basic reconstruction is obtained by solving the least-squares problem. A sparse reconstruction is then obtained by solving a L1-regularized problem. This reconstruction is used to initialize the dictionaries. In the second stage, the estimated signal, reconstructed target, and dictionaries are updated iteratively to obtain the final reconstruction.

Schematic of NLOS layout and the proposed SOCR reconstruction algorithm. a To reconstruct the unseen object, some light sources are used to illuminate a visible wall. The photons bounced back from the object are detected at several points on the visible wall. b Ground truth and our reconstruction. The x, y and z components are displayed in their absolute values. c Flowchart of the proposed framework. The estimated signal and the learned patterns of the target are shown on the left. Reconstructed albedo is shown on the right.

In summary, we have proposed a signal-object collaborative regularization (SOCR) framework for high-quality reconstruction of the albedo and surface normal of the unseen target. The method is robust to noise and applicable in both confocal and non-confocal settings. It can be used as a plug-in module in different physical models. 

More details can be found in our article published in Light: Science & Applications (

Lingyun Qiu

Assistant Professor, Tsinghua University