Learning to Propagate Information Across Pixels

  • Varun Jampani | Institute for Intelligent Systems (MPI) in Tübingen, Germany

Natural images exhibit high information correlation across pixels. Bilateral filtering provide a simple yet powerful framework for information propagation across pixels. The common use-case is to manually choose a parametric filter type, usually a Gaussian filter. In this work, we generalise the paramterization using a high-dimensional linear approximation and derive a gradient descent algorithm so the filter parameters can be learned from data. We demonstrate the use of learned bilateral filters in several diverse applications where Gaussian bilateral filters are traditionally employed: color upsampling, depth upsampling, semantic segmentation, material segmentation and 3D mesh denoising. We consistently observed improvements with filter learning. In addition, the ability to learn generic high-dimensional sparse filters allows us to stack several parallel and sequential filters like in convolutional neural networks (CNN) resulting in new breed of neural networks which we call ‘Bilateral Neural Networks’ (BNN). We demonstrate the use of BNNs using an illustrative segmentation problem and sparse character recognition.

In the later part of the talk, I will touch upon our recent work on segmentation CNNs where information propagation across pixels is generally performed as a post-processing step with either using conditional random fields (CRF) or up-convolution techniques. In contrast, we propose to propagate information across units inside the CNN itself with our newly introduced ‘Bilateral Inception’ modules. Experiments on different base segmentation networks and on different datasets showed that our bilateral inception modules result in reliable performance gains in terms of both speed and accuracy in comparison to traditionally employed dense-CRF / Deconvolution techniques and also recently introduced dense pixel prediction techniques.

Speaker Details

Varun Jampani is a final year doctoral student at Max Planck Institute for Intelligent Systems (MPI) in Tubingen, Germany. He works in the area of machine learning and computer vision with the supervision of Dr. Peter V. Gehler. His main research interests include probabilistic inference and neural networks. Before coming to MPI-Tubingen, he obtained his Bachelor of Technology and Master of Science degrees from International Institute of Information Technology, Hyderabad (IIIT-H), India, where he was a gold medalist. During his studies, he did internships at Microsoft research institutes in Redmond (US), Cambridge (UK) and Cairo (Egypt); MPI, Tubingen (Germany) and; GE global research, Bangalore (India). He also worked as a volunteer teacher in Tibetan Children’s Village, Dharamsala, India.

Watch Next