Shape from Polarization for Complex Scenes in the Wild

Abstract

We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image. Existing shape from polarization (SfP) works mainly focus on estimating the normal of a single object rather than complex scenes in the wild. A key barrier to high-quality scene-level SfP is the lack of real-world SfP data in complex scenes. Hence, we contribute the first real-world scene-level SfP dataset with paired input polarization images and ground-truth normal maps. Then we propose a learning-based framework with a multi-head self-attention module and viewing encoding, which is designed to handle increasing polarization ambiguities caused by complex materials and non-orthographic projection in scene-level SfP. Our trained model can be generalized to far-field outdoor scenes as the relationship between polarized light and surface normals is not affected by distance. Experimental results demonstrate that our approach significantly outperforms existing SfP models on two datasets. Our dataset and source code will be publicly available at https://github.com/ChenyangLEI/sfp-wild.

Files

Importance of polarization

• Passive sensing: the polarization camera is a passive sensor that is not constrained to a specific depth range.
• Dense geometry cues: the polarization images can provide dense surface orientation cues from the polarized light perceived at each pixel.
• Physics-based cues: the polarization cues are based on real-world reflections.

Physics-based cues

Note that polarization conveys the underlying physical shape. The estimated normal maps are not distrated by the semantics in a printed picture attached to a wall.

Input unpolarized intensity	Input polarization angle	RGB-based methods	Ours

Passive sensing

Although our model is trained on near-field content, it appears to successfully generalize to large-scale outdoor scenes.

Input unpolarized intensity	Ours

BibTeX