Microsoft Research Blog

Microsoft Research Blog

The Microsoft Research blog provides in-depth views and perspectives from our researchers, scientists and engineers, plus information about noteworthy events and conferences, scholarships, and fellowships designed for academic and scientific communities.

Microsoft HoloLens facilitates computer vision research by providing access to raw image sensor streams with Research Mode

June 18, 2018 | By Marc Pollefeys, Partner Director of Science

Microsoft HoloLens is the world’s first self-contained holographic computer. Remarkably, in Research Mode, available in the newest release of Windows 10 for HoloLens, it’s also a potent computer vision research device. Application code can not only access video and audio streams but can also at the same time leverage the results of built-in computer vision algorithms such as SLAM (simultaneous localization and mapping) to obtain the motion of the device as well as the spatial-mapping algorithms to obtain 3D meshes of the environment. These capabilities are made possible by several built-in image sensors that complement the color video camera normally accessible to applications.

Specifically, HoloLens has four gray-scale environment tracking cameras and a depth camera to sense its environment and capture gestures of the user. As shown in Figure 1, two of the gray-scale cameras are configured as a stereo rig capturing the area in front of the device so that the absolute depth of tracked visual features can be determined through triangulation. Meanwhile the two additional gray-scale cameras help provide a wider field of view to keep track of features. These synchronized global-shutter cameras are significantly more light-sensitive than the color camera and can be used to capture images at a rate of up to 30 frames per second (FPS).

The depth camera uses active infrared (IR) illumination to determine depth through time-of-flight. The camera can operate in two modes. The first mode enables high-frequency (30 FPS) near-depth sensing, commonly used for hand tracking, while the other is used for lower-frequency (1-5 FPS) far-depth sensing, currently used by spatial mapping. In addition to depth, this camera also delivers actively illuminated IR images that can be valuable in their own right because they are illuminated from the HoloLens and reasonably unaffected by ambient light.

Figure 1 – The HoloLens additional built-in gray scale image sensors complement the color camera normally available to applications.

With the newest release of Windows 10 for HoloLens, researchers now have the option to enable Research Mode on their HoloLens devices to gain access to all these raw image sensors streams, shown in Figure 2. Researchers can still use the results of the built-in computer vision algorithms but can now also choose to use the raw sensor data for their own algorithms. The sensors’ streams can either be processed on device or transferred wirelessly to another PC or to the cloud for more computationally demanding tasks. This opens a wide range of new computer vision applications for HoloLens. In egocentric vision, HoloLens can be used to analyze the world from the perspective of a user wearing the device. For these applications, HoloLens abilities to visualize results of the algorithms in the 3D world in front of the user can be a key advantage. HoloLens sensing capabilities can also be very valuable for robotics where these can, for example, enable a robot to navigate its environment.

Figure 2 – A sample HoloLens application that displays any of the Research Mode streams in real time.

These new HoloLens capabilities will be demonstrated at a tutorial on June 19th, 2018, at the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) in Salt Lake City. The next generation HoloLens depth sensing capabilities, which will also be made available through Project Kinect for Azure, will also be demonstrated at this tutorial.

We hope to see you there!

Learn more:
HoloLens Research Mode documentation
HoloLens Research Mode session at CVPR 2018
Alex Kipman’s Project Kinect for Azure blog on LinkedIn
Register your interest in Project Kinect for Azure

Up Next

mountains at ECCV

Computer vision, Data management, analysis and visualization

Computer Vision at Microsoft: Uniting fundamental research and industry-defining products

Microsoft is very proud to be a diamond sponsor of ECCV 2018 and we’re in Munich, Germany from September 8-14 with the global computer vision community to share our research and to learn from our fellow contributors. At Microsoft, in parallel with fundamental research, we build products. Our software products, like Visual Studio, PowerPoint and […]

Andrew Fitzgibbon

Partner Scientist, HoloLens

Artificial intelligence, Computer vision, Graphics and multimedia

Second version of HoloLens HPU will incorporate AI coprocessor for implementing DNNs

By Marc Pollefeys, Director of Science, HoloLens It is not an exaggeration to say that deep learning has taken the world of computer vision, and many other recognition tasks, by storm. Many of the most difficult recognition problems have seen gains over the past few years that are astonishing. Although we have seen large improvements […]

Microsoft blog editor

Data management, analysis and visualization, Graphics and multimedia

Partnership yields key breakthroughs in VR’s “grand challenge”

By Noboru Sean Kuno, Research Program Manager, Microsoft Research Asia The potential for virtual reality (VR) to upend industrial design, medicine, and other specialized fields has now vaulted the emerging field into the ranks of what the National Academy of Engineering calls its 14 grand challenges of the 21st century, an eclectic list of endeavors […]

Microsoft blog editor