Portrait of Pushmeet Kohli

Pushmeet Kohli

Principal Researcher

About

My research concerns the development of intelligent machines – to “teach” computers to (1) understand the behaviour and intent of humans, and (2) to correctly interpret (“Perceive” or “See”) objects and scenes depicted in colour/depth images or videos. I work in the areas of Computer Vision (RGB and 3D – KINECT), Machine Learning, Discrete Optimization, Behavioural Game Theory and Human-Computer Interaction. My current research interests include 3D Reconstruction and Rendering, Probabilistic Programming, Interpretable and Verifiable Knowledge Representations from Deep Models. In terms of applications, I am interested in Conversation agents for Task completion, Machine learning systems for Healthcare and 3D rendering and interaction for augmented and virtual reality.

In May 2015, I moved to Microsoft Redmond where I am acting as the Machine Learning advisor to the Chief Research Officer, Rick Rashid.

Projects

SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips

Established: June 29, 2015

We present a new interactive approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment, whilst interactively segmenting the scene simply by reaching out and touching any desired object or surface. Our system continuously learns…

Learning to be a depth camera for close-range human capture and interaction

Established: July 14, 2014

We present a machine learning technique for estimating absolute, per-pixel depth using any conventional monocular 2D camera, with minor hardware modifications. Our approach targets close-range human capture and interaction where dense 3D estimation of hands and faces is desired. We…

Filter Forests for Learning Data-Dependent Convolutional Kernels

Established: February 10, 2014

We propose ‘filter forests’ (FF), an efficient new discriminative approach for predicting continuous variables given a signal and its context. FF can be used for general signal restoration tasks that can be tackled via convolutional filtering, where it attempts to…

KinectFusion Project Page

Established: August 9, 2011

This project investigates techniques to track the 6DOF position of handheld depth sensing cameras, such as Kinect, as they move through space and perform high quality 3D surface reconstructions for interaction. Other collaborators (missing from the list below): Richard Newcombe…

Publications

2015

2014

2013

2012

2011

2010

2009

2008

2005

Projects

Other

Home

Some Recent Projects

  • Deep Interpretable Models for Visual and Conversational Data
  • Probabilistic Programming for Perception
  • Generative Models for Adaptive Crowdsourcing and Aggregation
  • Non-greedy Training of Decision/Regression Forests
  • Scene Understanding (MobileFusion, Semantic Paint, Segmentation, Detection, Geometry)
  • Human Pose Estimation for Kinect and RGB cameras
  • KinectFusion: Real time 3D Reconstruction using the Kinect and Image and Video Editing
  • Behavioural Game Theory via Research Games

Academic and Industrial Duties

Recent News

  • May 2016, New work on Hand pose estimation will appear in SIGGRAPH 2016.
  • May 2016, Our paper on real time full body reconstruction (Fusion 4D) will appear in SIGGRAPH 2016.
  • March 2016, We have developed a super efficient algorithm (Global Patch Collider) for correspondece estimation that will appear in CVPR 2016.
  • March 2016, Paper on Layered Scene Decomposition via the Occlusion-CRF will be presented at CVPR 2016.
  • February 2016, Our new Visual storytelling dataset and its accompanying paper will be released at NAACL 2016.
  • February 2016, Cloze Test – our proposal for evaluating the ability to cature narrative structure – to be released in our NAACL 2016 paper.
  • January 2016, Moving to the Microsoft headquaters in Redmond to start a new AI team.
  • December 2015, Paper on Non-greedy training of decision trees appears in NIPS 2015.
  • December 2015, Paper on interpretable autoencoders – Deep Convolutional Inverse Graphics Network is presented at NIPS 2015.
  • September 2015, Paper on Hierarchical Sampling Optimization for Estimating Human Hand Pose will appear at ICCV 2015.
  • August 2015, MobileFusion has received a lot of press converage (see articles in (WIRED, Mashable, and even the Register)
  • August 2015, Our paper Mobile Fusion that describes a method for generating 3D reconstruction of objects using a standard smartphone has been accepted to appear in ISMAR 2015.
  • July 2015, Our paper on Maximum Flows by Incremental Breadth-First Search (with Andrew Goldberg, Sagi Hed, Robert Tarjan, Renato Werneck and Haim Kaplan) will appear at the European Symposium on Algorithms (ESA)
  • July 2015, Interactive Scene Understanding (Semantic Paint) is covered widely in the Press (BBC, Engadget, and even the Daily Mail).
  • July 2015, Our paper on learning to decipher heaps for software verificiation wins the Best Paper Award at the Constructive ML workshop at ICML 2015.
  • June 2015, Our paper on “Interactive Scene Understanding” – a collaboration with University of Oxford and Stanford will be presented at SIGGRAPH 2015. See video here.
  • June 2015, The “Picture Probablitic Programming” paper receives the Best Paper Honorable Mention Award at CVPR 2015.
  • May 2015, Paper on learning Perturbation Models with Multidimensional Parametric Min-cuts has been accepted to appear in UAI 2015.
  • May 2015, Paper on Information Gathering in Networks via Active Exploration has been accepted to appear in IJCAI 2015.
  • May 2015, Video of the Hand Pose Estimation work is now available here.
  • April 2015, Picture, a new programming language for vision problems, has been accepted to appear in CVPR 2015 and has received a lot of press coverage (MIT News, The Register, Scientific Computing, iProgrammer).
  • April 2015, Paper on Real-time Hand Pose Estimation is appearing in CHI 2015.
  • March 2015, Our paper on fast hashing has been accepted to appear in CVPR 2015.
  • February 2015, Paper on Crowdsourcing Language Understanding in the Wild is accepted to appear in WWW 2015. See blog posts here and here.
  • January 2015, Paper on Consensus Message Passing is accepted to appear in AISTATS 2015
  • December 2014, Joined the Advisory board for the NEMOG (New Economic Models for Digital Games) project.
  • November 2014, Paper on Learning with Multiple Annotation-specific Loss Functions is accepted to appear in EMMCVPR 2015.
  • October 2014, Paper and Video describing our work on automatic layout of virtual objects for augmented reality released at ISMAR 2014.
  • October 2014, Paper on Just-in-Time Inference is accepted to appear in NIPS 2014.
  • September 2014, Paper on Real-time Face Reconstruction from a Single Depth Image is accepted to appear in 3DV 2014.
  • August 2014, Our paper on Exploration of Group Viewing Patterns Paper is the Runner-up for the Best Paper Award at TVX 2014.
  • July 2014, 3 papers (contour completion, learning with perceptual loss functions, Non-parametric Higher-order MRFs) are accepted to appear in ECCV 2014.
  • June 2014, Our demo on FilterForest for Image Denoising is the Runner-up for the Best Demo Award at CVPR 2014.
  • May 2014, Paper on automatic layout of virtual objects for augmented reality is accepted to appear at ISMAR 2014.
  • April 2014, Paper on community priors for crowdsourcing is the Runner-up for the Best Paper Award at WWW 2014.
  • March 2014, Paper on Depth from IR illumination fall-off is accepted to appear in SIGGRAPH 2014.
  • February 2014, Our edited volume Tractability: Practical Approaches to Hard Problems is released by Cambridge University Press.
  • February 2014, 3 Papers (Learning Portfolios for camera relocatlization, Personalized gesture recognition, Filter Forests for Image Labelling) are accepted to appear in CVPR (2 for oral presentation).
  • January 2014, Paper on encoraging diversity in multiple-output prediction is accepted for oral presentation at AISTATS 2014.
  • January 2014, Paper on community priors for crowdsourcing is accepted to appear in WWW 2014.
  • December 2013, Our paper on the effect of principles on power of agents in strategic games is accepted to appear in AAMAS 2014.
  • December 2013, Paper on User Behaviour Adaptation Under Interface Change accepted to appear in IUI 2014.
  • November 2013, Our Infer.Net based model for crowdsourcing Won the CrowdScale challenge at HCOMP 2013.
  • October 2013, Gave talk at the Human Behaviour Understanding workshop at ACM multimedia.
  • September 2013, Paper on Decision DAGs accepted to appear in NIPS 2013.
  • September 2013, Invited to join the editorial board of CVIU.
  • August 2013, Paper on Sementic labelling of Voxel Spaces accepted to appear in ICCV 2013.
  • July 2013, Paper on text detection and recognition accept to appear at ICDAR 2013.
  • June 2013, Tutorial on Solving real world problems with RGBD sensors at CVPR 2013.
  • June 2013, Gave a talk at the London School of Economics (LSE) on the value of Big Data.
  • April 2013, Papers on online algorithms for diverse recommendations and computation of coalition structures in coalition games are accepted to appear in AAAI 2013.
  • March 2013, Four papers (two for oral presentation) accepted to appear in CVPR 2013.
  • February 2013, Invited to join the editorial board of IJCV.
  • January 2013, Paper on faster training of structural SVMs is accepted to appear in AISTATS 2012 ( video ).
  • October 2012, PAMI paper describing the pose estimation system for KINECT has been accepted.
  • September 2012, Two papers (1) structured output prediction with multiple choices (2) context driven random forests are accepted to appear in NIPS 2012.
  • August 2012, The accepted ECCV and DAGM papers are now available online.
  • July 2012, Jamie Shotton and I have written a new book chapter on our recent work on Human Pose Estimation for the KINECT.
  • June 2012, Five papers (two for oral presentation) accepted to appear in the European Conference on Computer Vision (ECCV).
  • May 2012, The MSR KINECT gesture dataset collected for our CHI 2012 paper is now available.
  • April 2012, 3 Papers on Behavioural Game Theory and Personality-Online Behaviour patterns are accepted to appear in ACM WebSci 2012 .
  • March 2012, Papers accepted to CHI 2012, AISTATS 2012, Eurographics 2012 and PAMI/IJCV are now available online.
  • January 2012, Carsten Rother and I are teaching the Advanced Computer Vision course in the Engineering department of the University of Cambridge.
  • October 2011, Our paper on simulataneous localization and 3D mapping wins the Best Paper Award at ISMAR 2011.
  • October 2011, Interviews on the role of Behavioural game theory appear in Forbes and The Economic Times.
  • September 2011, Our first Facebook app (MSR Project Waterloo) for conducting behavioural game theory experiments is now online!
  • September 2011, The KinectFusion system has been made public. (Project Page) (video) (See UIST and ISMAR papers for detail)
  • June 2011, Our book on Advances in Markov Random Fields for Vision and Image Processing has been published by MIT press.
  • June 2011, 4 papers on adpative algorithms for MAP inference appear in AISTATS 2011, ICML 2011 and CVPR 2011.
  • December 2011, Our paper on evaluation and learning of interactive Segmentation systems wins the Best Paper Award at ICVGIP 2010.
  • October 2010, Our paper on inference with co-occurence potential wins the Best Paper Award at ECCV 2010.

Books and Edited Volumes

News

News

  • September 2015, Our paper on Non-greedy training of decision trees will appear in NIPS 2015.
  • September 2015, Our paper on interpretable autoencoders – Deep Convolutional Inverse Graphics Network will appear in NIPS 2015.
  • August 2015, MobileFusion has received a lot of press converage (see articles in (WIRED, Mashable, and even the Register)
  • August 2015, Our paper Mobile Fusion that describes a method for generating 3D reconstruction of objects using a standard smartphone has been accepted to appear in ISMAR 2015.
  • July 2015, Our paper on Maximum Flows by Incremental Breadth-First Search (with Andrew Goldberg, Sagi Hed, Robert Tarjan, Renato Werneck and Haim Kaplan) will appear at the European Symposium on Algorithms (ESA)
  • July 2015, Interactive Scene Understanding (Semantic Paint) is covered widely in the Press (BBC, Engadget, and even the Daily Mail).
  • July 2015, Our paper on learning to decipher heaps for software verificiation wins the Best Paper Award at the Constructive ML workshop at ICML 2015.
  • June 2015, Our paper on “Interactive Scene Understanding” – a collaboration with University of Oxford and Stanford will be presented at SIGGRAPH 2015. See video here.
  • June 2015, The “Picture Probablitic Programming” paper receives the Best Paper Honorable Mention Award at CVPR 2015.
  • May 2015, Paper on learning Perturbation Models with Multidimensional Parametric Min-cuts has been accepted to appear in UAI 2015.
  • May 2015, Paper on Information Gathering in Networks via Active Exploration has been accepted to appear in IJCAI 2015.
  • May 2015, Video of the Hand Pose Estimation work is now available here.
  • April 2015, Picture, a new programming language for vision problems, has been accepted to appear in CVPR 2015 and has received a lot of press coverage (MIT News, The Register, Scientific Computing, iProgrammer).
  • April 2015, Paper on Real-time Hand Pose Estimation is appearing in CHI 2015.
  • March 2015, Our paper on fast hashing has been accepted to appear in CVPR 2015.
  • February 2015, Paper on Crowdsourcing Language Understanding in the Wild is accepted to appear in WWW 2015
  • January 2015, Paper on Consensus Message Passing is accepted to appear in AISTATS 2015
  • December 2014, Joined the Advisory board for the NEMOG (New Economic Models for Digital Games) project.
  • November 2014, Paper on Learning with Multiple Annotation-specific Loss Functions is accepted to appear in EMMCVPR 2015.
  • October 2014, Paper and Video describing our work on automatic layout of virtual objects for augmented reality released at ISMAR 2014.
  • October 2014, Paper on Just-in-Time Inference is accepted to appear in NIPS 2014.
  • September 2014, Paper on Real-time Face Reconstruction from a Single Depth Image is accepted to appear in 3DV 2014.
  • August 2014, Our paper on Exploration of Group Viewing Patterns Paper is the Runner-up for the Best Paper Award at TVX 2014.
  • July 2014, 3 papers (contour completion, learning with perceptual loss functions, Non-parametric Higher-order MRFs) are accepted to appear in ECCV 2014.
  • June 2014, Our demo on FilterForest for Image Denoising is the Runner-up for the Best Demo Award at CVPR 2014.
  • May 2014, Paper on automatic layout of virtual objects for augmented reality is accepted to appear at ISMAR 2014.
  • April 2014, Paper on community priors for crowdsourcing is the runner-up for the best paper award at WWW 2014.
  • March 2014, Paper on Depth from IR illumination fall-off is accepted to appear in SIGGRAPH 2014.
  • February 2014, 3 Papers (Learning Portfolios for camera relocatlization, Personalized gesture recognition, Filter Forests for Image Labelling) are accepted to appear in CVPR (2 for oral presentation).
  • January 2014, Paper on encoraging diversity in multiple-output prediction is accepted for oral presentation at AISTATS 2014.
  • January 2014, Paper on community priors for crowdsourcing is accepted to appear in WWW 2014.
  • December 2013, Our paper on the effect of principles on power of agents in strategic games is accepted to appear in AAMAS 2014.
  • December 2013, Paper on User Behaviour Adaptation Under Interface Change accepted to appear in IUI 2014.
  • November 2013, Our Infer.Net based model for crowdsourcing won the CrowdScale challenge at HCOMP 2013.
  • October 2013, Gave talk at the Human Behaviour Understanding workshop at ACM multimedia.
  • September 2013, Paper on Decision DAGs accepted to appear in NIPS 2013.
  • September 2013, Invited to join the editorial board of CVIU.
  • August 2013, Paper on Sementic labelling of Voxel Spaces accepted to appear in ICCV 2013.
  • July 2013, Paper on text detection and recognition accept to appear at ICDAR 2013.
  • June 2013, Tutorial on Solving real world problems with RGBD sensors at CVPR 2013.
  • June 2013, Gave a talk at the London School of Economics (LSE) on the value of Big Data.
  • April 2013, Papers on online algorithms for diverse recommendations and computation of coalition structures in coalition games are accepted to appear in AAAI 2013.
  • March 2013, Four papers (two for oral presentation) accepted to appear in CVPR 2013.
  • February 2013, Invited to join the editorial board of IJCV.
  • January 2013, Paper on faster training of structural SVMs is accepted to appear in AISTATS 2012 ( video ).
  • October 2012, PAMI paper describing the pose estimation system for KINECT has been accepted.
  • September 2012, Two papers (1) structured output prediction with multiple choices (2) context driven random forests are accepted to appear in NIPS 2012.
  • August 2012, The accepted ECCV and DAGM papers are now available online.
  • July 2012, Jamie Shotton and I have written a new book chapter on our recent work on Human Pose Estimation for the KINECT.
  • June 2012, 5 Papers accepted to appear in the European Conference on Computer Vision (ECCV).
  • May 2012, The MSR KINECT gesture dataset collected for our CHI 2012 paper is now available.
  • April 2012, 3 Papers on Behavioural Game Theory and Personality-Online Behaviour patterns are accepted to appear in ACM WebSci 2012 .
  • March 2012, Papers accepted to CHI 2012, AISTATS 2012, Eurographics 2012 and PAMI/IJCV are now available online.
  • Februray 2012, New Facebook game Doubloon Dash designed to study strategies used in all-pay auctions is now available on Facebook .
  • January 2012, Carsten Rother and I are teaching the Advanced Computer Vision course in the Engineering department of the University of Cambridge.
  • December 2011, Videos describing some of my research can be found here.
  • November 2011, Selected to join ACM’s Distinguished Speaker Program.
  • October 2011, Our paper on simulataneous localization and 3D mapping wins the Best Paper Award at ISMAR 2011.
  • October 2011, Interviews on the role of Behavioural game theory appear in Forbes and The Economic Times.
  • September 2011, Our first Facebook app (MSR Project Waterloo) for conducting behavioural game theory experiments is now online!.
  • September 2011, Kinect fusion gets reviewed by MIT Technology review (full article).
  • September 2011, The KinectFusion system has been made public. (Project Page) (video) (See UIST and ISMAR papers for detail)
  • September 2011, The ICCV papers on Decision Tree Fields and Regression for Human Pose Estimation are now online.
  • June 2011, Our book on Advances in Markov Random Fields for Vision and Image Processing has been published by MIT press.
  • June 2011, Slides for the invited tutorial at IBPRIA are now online! (See Invited Tutorial on MAP Inference in Discrete Models)
  • June 2011, 4 papers on improving speed and accuracy of conventional algorithms for MAP inference by making them energy or problem-aware have appeared in AISTATS, ICML and CVPR 2010.
  • December 2011, Our paper on evaluation and learning of interactive Segmentation systems wins the Best Paper Award at ICVGIP 2010.
  • October 2010, Our paper on inference with co-occurence potential wins the Best Paper Award at ECCV 2010.

Research

Research

My research concerns the development of intelligent machines, with the particular aim of “teaching” computers to (1) understand the behaviour and intent of human users, and (2) to correctly interpret (or “See”) objects and scenes depicted in colour/depth images or videos. I work and publish in the fields of Computer Vision, Machine Learning, Discrete Optimization, Game Theory and Human-Computer Interaction.

My current research interests include 3D Reconstruction and Rendering, Probabilistic Programming, Interpretable and Verifiable Knowledge Representations from Deep Models. In terms of real world applications, I am particularly interested in Conversation agents for Task completion, Machine learning systems for Healthcare and 3D rendering and interaction for augmented and virtual reality.

Computer Vision

  • Structured Representations for Visual Knowledge and Commonsense
  • Low-level vision problems: Image Segmentation, Dense Stereo, Optical Flow
  • Object Recognition and Segmentation
  • Human Pose Estimation from KINECT
  • Localization and Reconstruction using KINECT

Machine Learning

  • Verifiable and Interpetable Models
  • Probablistic Programming
  • MAP Inference in Discrete Models (Discrete Optimization)
  • Structured Learning
  • Learning of Interactive Systems

Game Theory

  • Behavioural game theory research using social networks such as Facebook
  • Finding Optimal Coalitions in Cooperative Games
  • Reconstructing Coalitional Games
  • Computing Optimal Coalition Structures

Information Retrieval

  • Personalizing Search
  • Psycho-metric profiles for capturing user intent

In past life, I have also dabbled a bit in model based checking of non-deterministic software systems. Some of my work can be found in Spec Explorer.

Collaborators

Past and Current Students and Interns

  • Lubor Ladicky, PhD student 2007-2011, (now Post-doc at Oxford)
  • Dhruv Batra, Intern 2010 (now Asst. Professor at TTI Chicago)
  • Michal Kosinski, Intern 2010 (now at Psychometrics Centre, Cambridge)
  • Patrick Pletscher, Intern 2010 (PhD student at ETH)
  • Bangpeng Yao, Intern 2010 (PhD student at Stanford)
  • Olga Barinova, Intern 2009 (now at Moscow State University)
  • Hannes Nickish, Intern 2009 (now at Philips Research)
  • Sara Vicente, Intern 2008
  • Dheeraj Singaraju, Intern 2008
  • Kyomin Jung, Intern 2008 (now Asst. Professor at KAIST)

Other Collaborators

  • Shahram Izadi, MSR Cambridge
  • Yoram Bachrach, MSR Cambridge
  • Thore Graepel, MSR Cambridge
  • Jamie Shotton, MSR Cambridge
  • Sebastian Nowozin, MSR Cambridge
  • Carsten Rother, MSR Cambridge
  • Andrew Fitzgibbon, MSR Cambridge
  • Otmar Hilliges, MSR Cambridge
  • Philip Torr, Oxford Brookes University
  • M Pawan Kumar, Ecole Centrale Paris
  • Richard Newcombe, Imperial
  • Chang Yoo, KAIST
  • More extended list at the DBLP site
  • Dear collaborator, please send me a mail if your name is missing from this list and you would want me to add it here.

Press Coverage

Press Coverage

Reconstructing and Depth Sensing from Standard Smartphones

Probabilistic Programming for Visual Perception

Semantic Understanding of 3D Spaces

Behavioral Game Theory

Kinect Fusion – Generating 3D Reconstructions

Unwrap Mosaics (Next-Gen Video Editing)

Professional Duties

Academic Duties

Program Committee Member – Reviewing Duties

  • Journal of Machine Learning Research (JMLR)
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
  • International Journal of Computer Vision (IJCV)
  • Computer Vision and Image Understanding (CVIU)
  • Neural Information Processing Systems (NIPS)
  • ACM SIGGRAPH
  • IEEE International Conference on Computer Vision (ICCV)
  • IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)
  • European Conference on Computer Vision (ECCV)
  • International Conference on Artificial Intelligence and Statistics(AISTATS)
  • British Machine Vision Conference (BMVC)

Tutorials

Invited Talks / Courses / Tutorials

Resources

Datasets

  • Simon Fothergill, Helena M. Mentis, Pushmeet Kohli, Sebastian Nowozin MSRC-12 Kinect gesture data set (2012) [Data]
  • Nathan Silberman, Pushmeet Kohli, Derek Hoiem, Rob Fergus NYU Depth Dataset V2 (2012) [Project Page]

Code

  • Pushmeet Kohli, Anton Osokin, Stefanie Jegelka Inference in Deep Random Field Model for Image Segmentation Using Graph Cuts, CVPR 2013 [MATLAB Code from Anton Osokin]
  • Pushmeet Kohli, Philip Torr Computing Min-marginal Using Graph Cuts, ECCV 2006 [MATLAB Code from Sara Vicente]
  • Danny Tarlow, Dhruv Batra , Pushmeet Kohli and Vladimir Kolmogorov Dynamic Tree Block Coordinate Ascent (2011) [zip]
  • Alex Rav-Acha, Pushmeet Kohli, Carsten Rother and Andrew Fitzgibbon Embedding Code for Unwrap Mosaics (2008) [Code]
  • Lubor Ladicky, Pushmeet Kohli and Philip Torr Minimizing Robust Higher Order Potentials using Move Making Algorithms (2008) [zip]
  • Karteek Alahari, Pushmeet Kohli and Philip Torr Reduce, Reuse and Recycle (2008) [zip][ReadMe]
  • Pushmeet Kohli and Philip Torr Dynamic Graph Cuts (2005) [zip]