Non-line-of-sight (NLOS) imaging can reconstruct or recognize hidden objects, and therefore holds potential in a wide range of applications like automatic vehicles, robotic vision, and biomedical imaging. The vast majority of NLOS imaging is based on active detection strategies requiring a fast pulsed or continuous laser, such as light detection and ranging (LIDAR), correlation-based imaging, and holographic approaches. However, due to the need for a simplified hardware system and good stealthiness, it is essential to adopt passive NLOS imaging in some practical scenarios. Passive NLOS imaging generally reveals worse reconstruction quality limited by the ultra-weak scattered signal especially when the prior information of the setting is unknown. Under this condition, passive NLOS recognition using convolutional neural network (CNN) is preferred over reconstruction owning to its ability to classify the hidden objects directly and quickly. However, to our knowledge, passive NLOS recognition has not been systematically investigated so far.

Our group at Qian Xuesen Laboratory of Space Technology performed passive NLOS recognition using a pin-speck experimental setup, which consists of a standard digital camera and an occluder with unknown shapes (Figure 1). The signal of interest is as low as 0.3 Lux measured in front of the camera lens. We test the NLOS recognition system on four increasingly complex scenes. For uncalibrated setups placed in a dark room, the recognition accuracy exceeds 97% on the well-known MNIST dataset of handwritten digits with less than 1 second per image. The proposed NLOS recognition system is also generalized on more sophisticated datasets of hand gestures, human postures, and fashion-MNIST. The recognition accuracy varies from 81.58% to 93.95%. When a homogeneous ambient light of 4.2 Lux is added to the setup, the recognition accuracy still maintains above 94% on the MNIST handwritten digit dataset. Then, we move the system in an exhibition hall and place an L-shaped translucent acrylic plate around the system. Ambient light is as high as 120 Lux measured in front of the camera lens. The recognition accuracy is 87.4% with data augmentation when no one around the system, which is considerably high considering the weak signal (0.3 Lux) and camera noise. When varying ambient light is added to the setup by using 0-3 plates to cast shadows on the secondary surface, the recognition accuracy remains above 88.28% using a retrained CNN model with the data collected under a mixed lighting condition. Moreover, the system reveals good generalizability with an accuracy above 60% when a different number of people walk around the system and cast shadows on the secondary surface. However, we had to admit that when the ambient light is intense of 610 Lux (without the L-shaped translucent acrylic plate), the system is invalid. We expect that other approaches like using polarization signal or multi-band fusion like thermal and visible images could be combined to further improve the robustness of the NLOS recognition system against the intense background light.

Figure 1. Schematic of the occluder-based passive Non-line-of-sight (NLOS) recognition system.

Conversely, we further consider how to conduct attacks on passive NLOS recognition. CNNs are vulnerable to attacks in line-of-sight (LOS) image classification, which limits their use in this domain. Whether are CNNs fragile in NLOS recognition? There is no consensus in the literature regarding this topic. The means to realize NLOS recognition attacks is challenging because the relationship between the image on the monitor and the captured penumbra on the secondary surface is not clear in uncalibrated setups. We perform both simulations of attacks and real-world attacks in this study. Finally, an attack success rate of approximately 36% is achieved in a real-world attack with a relatively low cost even using uncalibrated settings. Although the attack success rate currently is not as high as that in LOS recognition, we sincerely hope our work could inspire researchers to provide more strategies on NLOS anti-recognition.

To learn more about our work, please read our article "Accurate but Fragile Passive Non-Line-of-Sight Recognition" in  Communications Physics; https://www.nature.com/articles/s42005-021-00588-2.