Microsoft Research https://www.microsoft.com/en-us/research Tue, 23 May 2017 17:05:58 +0000 en-US hourly 1 https://wordpress.org/?v=4.7.5 Holograms: The future of near-eye display? https://www.microsoft.com/en-us/research/blog/holograms-future-near-eye-display/ Fri, 19 May 2017 21:45:31 +0000 https://www.microsoft.com/en-us/research/?p=385043 By Andrew Maimone, Researcher; Andreas Georgiou, Researcher; Joel Kollin, Principal Research Hardware Development Engineer Last week at the SCIEN Workshop on Augmented and Mixed Reality, a group of industry and academic researchers met to discuss the future of virtual and mixed reality (VR/MR). One goal was clear: we’d all like to put on devices that look and feel […]

The post Holograms: The future of near-eye display? appeared first on Microsoft Research.

]]>
By Andrew Maimone, Researcher; Andreas Georgiou, Researcher; Joel Kollin, Principal Research Hardware Development Engineer

Last week at the SCIEN Workshop on Augmented and Mixed Reality, a group of industry and academic researchers met to discuss the future of virtual and mixed reality (VR/MR). One goal was clear: we’d all like to put on devices that look and feel like ordinary eyeglasses but can bring us to new places. We also want to maintain the full fidelity of human vision while experiencing VR/MR.

We’re still in the early days on the journey toward this vision, and there isn’t a clear route to solving all the optical challenges. For example, today’s near-eye displays feature a trade-off between bulkiness and field of view. They provide only part of the information the brain uses to judge depth. It’s not clear how to make a display that looks like eyeglasses when many must wear actual eyeglasses to see clearly. Typically, when we try to solve one of these challenges, we take a step back in another area.

In a new publication to be presented at SIGGRAPH 2017, we propose a new solution to VR/MR displays that has the potential to address all these problems at once: holography. When light waves from a 3D scene meet on a flat surface, they interact to form a sophisticated pattern, known as a hologram — much like a series of waves combining to form a detailed set of ripples in a pool. The hologram can then be “played back” with a beam of laser light to re-create all the intricacies of the 3D scene, including the depths of all the objects.

One of the greatest challenges to building near-eye displays is balancing the trade-off between the field of view and device form factor. A wide field of view is desirable so that we can see the whole scene at once; at the same time, we desire a lightweight, compact and aesthetically pleasing design that is both comfortable to wear all day and socially acceptable. Today we find these qualities only in isolation: sunglasses-like displays with very narrow fields of view (<20 degrees horizontal), or goggle- or helmet-sized displays with wide fields of view (>= 80 degrees horizontal). In our work, we created a simple prototype display with a sunglasses-like form factor, but a wide field of view (80 degrees horizontal), shown below. The key insight from our admittedly crude prototype is that we can create an optical design with visual defects, that is very compact, and then correct the defects in the hologram calculation. In the image below, we show a photograph of our prototype display as well as a simple augmented image shot through the display — note the image is clear edge-to-edge.

Holograms: The future of near-eye display

Another key challenge in building near-eye displays is reproducing the focal depth cues. When viewing real objects, our eyes can focus on different depths, and the blurriness of out-of-focus objects is used to judge depth. However, most near-eye displays present all objects at the same focus. This limitation causes visual discomfort, reduces the realism of the scene and can cause focus conflicts between real and virtual objects. Many solutions have been proposed to address the problem — varifocal displays, multifocal displays and light-field displays — however, these solutions provide only a coarse approximation of the focal cues of the real scene. Holographic displays can provide highly accurate focal control that is individually addressable on a pixel by pixel basis, like a real scene.

In the image below, we show a hologram on a prototype display where each pixel in the image is at exactly the correct focus. The camera is focused at the dragon’s chest region, which appears in focus, while the nearer and farther regions appear out of focus. In the blue-bordered inset images, we show additional images of the chest, middle body and tail regions when the camera was focused at that region and focused away from that region. See the video for an animation of the camera focusing through the depth of the model.

Holograms: The future of near-eye display

Another important challenge to solve in near-eye displays is handling problems with the user’s vision. If we ultimately wish to make a display the size of eyeglasses, we must build the functionality of eyeglasses into the display. A current solution is to provide adjustable image focus control by mechanically moving the distance between the display panel and the lens. However, this solution does not scale to providing a display the size of eyeglasses, and does not correct for more advanced vision problems such as astigmatism. We demonstrate that holographic displays can correct for advanced vision problems entirely in software by pre-distorting the emitted light waves so that they appear correct when viewed by the eye.

In the image below, we demonstrate a vision correction capability on a prototype display. In the left image, for reference, we show a holographic display viewed with normal vision. In the center image, we show the display as it would look to a viewer with astigmatism — note in the blue-bordered inset image that the horizontal lines become blurred vertically. In the right image, we show the display viewed with astigmatism, but with compensation in the hologram; note that it looks virtually the same as when viewed with normal vision.

Holograms: The future of near-eye display

For more information, please visit our project page and video.

The Microsoft Research paper is not necessarily indicative of any Microsoft product roadmap, but relates to basic research around holographic displays.

Related

The post Holograms: The future of near-eye display? appeared first on Microsoft Research.

]]>
P: A programming language designed for asynchrony, fault-tolerance and uncertainty https://www.microsoft.com/en-us/research/blog/p-programming-language-asynchrony/ Fri, 19 May 2017 16:00:49 +0000 https://www.microsoft.com/en-us/research/?p=384782 By Shaz Qadeer, Principal Researcher The complexity of emerging software applications requires new approaches to understanding and then efficiently building, testing and debugging these systems. Today’s software uses cloud resources, is often embedded in devices in the physical world and employs artificial intelligence techniques. These three factors make today’s software systems an order of magnitude […]

The post P: A programming language designed for asynchrony, fault-tolerance and uncertainty appeared first on Microsoft Research.

]]>
By Shaz Qadeer, Principal Researcher

P programming language designed for asynchrony, fault-tolerance and uncertainty

The complexity of emerging software applications requires new approaches to understanding and then efficiently building, testing and debugging these systems. Today’s software uses cloud resources, is often embedded in devices in the physical world and employs artificial intelligence techniques. These three factors make today’s software systems an order of magnitude more difficult to develop.

Often, these modern applications feature asynchrony, which happens when, to improve performance, the requestor of an operation continues without requiring the completion of the operation. Asynchrony inevitably leads to concurrency, with its notorious pitfalls of race conditions and Heisenbugs (software bugs, often timing-related, that seems to disappear when investigated, due to changes resulting from the investigation). To address the challenges of asynchronous computation, we have developed P, a programming language for modeling and specifying protocols in asynchronous event-driven applications. This project is a collaborative effort between Microsoft researchers and engineers, and academic researchers at the University of California, Berkeley and Imperial College in London.

A flowchart of the P programming language tool chain

A flowchart of the P tool chain

The P programmer writes the protocol and its specification at a high level. The P compiler provides automated testing for concurrency-related race conditions and executable code for running the protocol. P provides first-class support for modeling concurrency, specifying safety and liveness properties and checking that the program satisfies its specification using systematic search. In these capabilities, it is similar to Leslie Lamport’s TLA+ and Gerard Holzmann’s SPIN. Unlike TLA+ and SPIN, a P program can also be compiled into executable C code. This capability bridges the gap between high-level model and low-level implementation and eliminates a huge hurdle to the acceptance of formal modeling and specification among programmers.

P programming language communicating state machines

Communicating state machines

The programming model in P is based on concurrently executing state machines communicating via events, with each event accompanied by a typed payload value. A memory management system based on linear typing and unique pointers provides safe memory management and data-race-free concurrent execution. In this respect, P is similar to modern systems programming languages such as Rust.

P got its start in Microsoft software development when it was used to ship the USB 3.0 drivers in Windows 8.1 and Windows Phone. These drivers handle one of the most important peripherals in the Windows ecosystem and run on hundreds of millions of devices today. P enabled the detection and debugging of hundreds of race conditions and Heisenbugs early on in the design of the drivers, and is now extensively used for driver development in Windows. Early positive experience with P in the Windows kernel led to the development of P#, a framework that provides state machines and systematic testing via an extension to C#. In contrast to P, the approach in P# is minimal syntactic extension and maximal use of libraries to deliver modeling, specification and testing capabilities.

P is transforming the development of cloud infrastructure in Azure. Azure, similar to other cloud providers, faces the challenge of Heisenbugs caused by unexpected race conditions and software or hardware faults. These bugs result in disruption of live services — a huge problem for both customers and providers of cloud services. P and P# are being used to find and debug Heisenbugs in already-deployed services and to design and validate new services before deployment. P allows engineers to precisely model asynchronous interfaces among components in a large Azure service. It also allows engineers to discover and debug problems on their desktops that would otherwise take months and sometimes even years to manifest after the service is deployed.

An important feature of P that makes it particularly suitable for validating fault-tolerant distributed services is the ability to perform thorough failover testing, that is, validating that the service recovers and resumes operation when an unexpected fault happens. Both network message drops and individual state machine failures are modeled as events. Modeling a fault as an event in P fully automates fault-injection and enables systematic testing of failover under a huge number of event orderings and faults, with little effort from the programmer.

The systematic testing capabilities in P thoroughly search over the choices induced by nondeterministic ordering of concurrently dispatched events. However, its capabilities are limited with respect to handling explicit data input, especially when that input is over large domains. This limitation makes it difficult to apply P to applications such as robotics where a significant source of complexity is decision-making under uncertain input. Dealing with large and uncertain input domains is an ongoing research investigation. We are investigating both symbolic and probabilistic techniques to deal with this challenge.

Download P

Download more from Microsoft’s P family of languages and tools

The post P: A programming language designed for asynchrony, fault-tolerance and uncertainty appeared first on Microsoft Research.

]]>
Where cryptography and quantum computing intersect https://www.microsoft.com/en-us/research/blog/cryptography-quantum-computing-intersect/ Mon, 08 May 2017 23:42:58 +0000 https://www.microsoft.com/en-us/research/?p=381917 By Kristin Lauter, Principal Researcher, Microsoft Research Last week I spent time at the American Institute of Mathematics in San Jose, working with a group of 20 or so mathematicians and computer scientists on questions related to quantum arithmetic, at a conference co-organized by researchers in the Microsoft Research (MSR) Quantum Architectures and Computation (QuArC) […]

The post Where cryptography and quantum computing intersect appeared first on Microsoft Research.

]]>
By Kristin Lauter, Principal Researcher, Microsoft Research

Last week I spent time at the American Institute of Mathematics in San Jose, working with a group of 20 or so mathematicians and computer scientists on questions related to quantum arithmetic, at a conference co-organized by researchers in the Microsoft Research (MSR) Quantum Architectures and Computation (QuArC) group. You might ask, why is a cryptographer working on this topic? (And also, when will we have a quantum computer? 🙂 )

I was at this workshop thanks to a surprising intersection between cryptography and quantum computing: An algorithm for breaking a cryptographic hash function which we found more than 10 years ago turns out to be relevant for designing efficient circuits for quantum arithmetic! In 2008, we published an efficient and constructive algorithm to find paths in LPS graphs to break the cryptographic hash function. The hash function was one in a family I had proposed in 2005 with Denis Charles (MSR) and Eyal Goren, professor, McGill University. Finding preimages for this hash function involved identifying paths in the well-known Lubotzky-Phillips-Sarnak (LPS) graphs, the celebrated Ramanujan (yes, The Man Who Knew Infinity) graphs constructed in the 1980s. This had been an open problem for several decades; we were delighted when we solved it.

But due to the stove pipes among research communities, our algorithm was not known to a group of researchers in quantum arithmetic, who wanted to solve this problem for a completely different reason. In 2014, Neil Ross and Peter Selinger published an algorithm that they found independently in a different setting, but which turns out to be essentially the same as our algorithm. They were trying to solve the problem of decomposing operations on a single qubit into quantum circuits and for finding the shortest possible circuit.

Researchers in the QuArC group at Microsoft Research, Vadym Kliuchnikov, Krysta Svore, Martin Roetteler, and others, had played a crucial role in developing a framework for addressing questions in quantum arithmetic more broadly, including for topological qubits, the focus of Microsoft’s approach to quantum computing (for Ising anyons, Fibonacci anyons, and metaplectic anyons).  This is important for the future of quantum computing because once a quantum computer is built, we will need to efficiently express operations on qubits in terms of fundamental gates that can be realized in the physical world.  One of the candidates identified are the Clifford+T gates.  The connection between our LPS path-finding algorithm and this area was discovered in 2015 by Alex Gamburd, professor, Graduate Center, CUNY, and Peter Sarnak, professor, Princeton University, and now, topics from both cryptography and quantum arithmetic are included in the workshop.

I have recently become interested in quantum computation for another reason that is important to the future of cryptography. The security of our e-commerce and cloud-based systems and the privacy of much of our data rely on the strength of currently deployed public key cryptographic systems such as RSA and Elliptic Curve Cryptography (ECC).  Once robust quantum computers can be built at a large enough scale, polynomial time quantum algorithms can be implemented to attack the underlying math problems for these cryptosystems, rendering them insecure. However, it is not clear how soon a sufficiently large and robust quantum computing device will be built, and also which architecture and gate set will be realizable. This makes it difficult to estimate the concrete security for cryptosystems of a given bit length in a post-quantum environment, and difficult to estimate the appropriate timeline for migration to new cryptosystems. In collaboration with researchers in QuArC, we have been developing concrete quantum estimates for breaking current encryption systems. At the workshop, we discussed various possible quantum gate sets that might become real; some of the project groups focused on developing the mathematics to implement efficient computation if those gate sets are built.

There may be hard instances of the “path-finding problem” for explicit choices of expander graphs, and such graphs could be the foundation for post-quantum cryptosystems, meaning classical protocols that are resistant to quantum attacks. This year, the National Institute of Standards and Technology (NIST) is running an international Post-Quantum Cryptography (PQC) competition, with submissions due in November, to determine possible replacements for currently deployed cryptographic systems. One of the possible candidates for standardization is based on the hard problem, path-finding in Supersingular Isogeny Graphs, introduced in our 2006 cryptographic hash function paper.  This paper was presented at an NIST workshop in 2005, a workshop leading to standardizing the new cryptographic hash function SHA-3, and was described in a news article in Science magazine in 2008.

In that paper we proposed the hard problem of finding paths or cycles in the Supersingular Isogeny Graph. This hard problem has since been used as the basis for proposals for encryption, key exchange and digital signatures, which are the three tracks of the NIST competition. Supersingular Isogeny Graphs consist of vertices, which are isomorphism classes of supersingular elliptic curves in characteristic p, and the edges are isogenies, or maps between curves that are homomorphisms, thus the name Supersingular Isogeny Graph. Originally studied and generalized by Arnold Pizer, professor emeritus, University of Rochester, these graphs have connections to deep theorems in number theory and beautiful properties such as optimal expansion constants. The expansion constants ensure that relatively short walks on the graph quickly approximate the uniform distribution in the output.

 

Supersingular Isogeny Graph

A small example of a Supersingular Isogeny Graph, for the prime p=2521, graph image by Denis Charles, principal applied scientist, Microsoft Research

Like many other public key systems, operations necessary to implement Supersingular Isogeny Graphs can be costly, so there is no guarantee that these systems would be competitive from a performance point of view. Also, like many of the other possible math-based candidates such as code-based cryptography, lattice-based cryptography and multivariate cryptosystems, relatively new math problems need to be studied for possible vulnerability to both classical and quantum algorithms. Thanks to the NIST PQC competition, we can expect mathematicians and cryptographers to be very busy during the next few years as we determine the best possible choices for our post-quantum cryptography standards.

Learn more

The post Where cryptography and quantum computing intersect appeared first on Microsoft Research.

]]>
CHI Part 2: Creating technologies that enhance lives in the areas of health and eye tracking https://www.microsoft.com/en-us/research/blog/chi-2-creating-technologies-enhance-lives-health-eye-tracking/ Mon, 08 May 2017 13:00:19 +0000 https://www.microsoft.com/en-us/research/?p=381518 By Justin Cranshaw, Researcher, Microsoft Research With its focus on the relationships between people and computation, the field of human-computer interaction (HCI) excels at creating technologies that marry human skills with computational functionality, to enable outcomes neither humans nor computers could achieve by themselves. This year at the ACM CHI Conference on Human Factors in […]

The post CHI Part 2: Creating technologies that enhance lives in the areas of health and eye tracking appeared first on Microsoft Research.

]]>
By Justin Cranshaw, Researcher, Microsoft Research

CHI 2017

With its focus on the relationships between people and computation, the field of human-computer interaction (HCI) excels at creating technologies that marry human skills with computational functionality, to enable outcomes neither humans nor computers could achieve by themselves. This year at the ACM CHI Conference on Human Factors in Computing Systems, Microsoft is presenting several ground-breaking works showcasing how humans and machines can work better together.

Building a healthier future

Recent advances in sensing, machine learning, and personal device design offer a promising vision for enhancing people’s lives in the most fundamental ways—by improving their health and well-being. HCI researchers at Microsoft have been exploring ways computers and people—healthcare providers and patients alike—can each work together in complementary ways to build a healthier future.

Multiple sclerosis (MS) is a debilitating neurological disease that afflicts millions of people worldwide. Because attacks can occur at unpredictable intervals over a long period, it is important for treatment regimens to include regular monitoring to track the progression of the disease. Assess MS is a system that uses the computer vision functionality of Microsoft’s Kinect to aid medical professionals in assessing the disease progression in MS. In a paper appearing at CHI; Assessing Multiple Sclerosis with Kinect: Designing Computer Vision Systems for Real-world Use, researchers, Cecily Morrison, Kit Huckvale, Robert Corish, Jonas F Dorn, Peter Kontschieder, Kenton O’Hara, the Assess MS Team, Antonio Criminisi, and Abigail Sellen cover the design process and resulting insights of a prototype system. This system enables a health professional and a computer vision system to work together on MS monitoring, using the natural strengths of the human and computer.

With the recent rise of technologies that gather and track personal healthcare data, it’s important to design to how patients and healthcare providers actually engage with these technologies. As reported in their paper—Self-tracking for Mental Wellness: Understanding Expert Perspectives and Student ExperiencesBongshin Lee, Microsoft senior researcher, along with Lauren Wilcox, assistant professor, and student Christina Kelley, both from Georgia Institute of Technology, conducted research to better understand how personal data about one’s mood, physical health, and social activities—collected on devices such as smartphones and activity trackers—can be used to help students manage stress and improve their mental wellness. By focusing on stress, anxiety and depression, these studies reveal how both students and health professionals prefer to engage with data and devices. This work will inform the design of applications, wearable devices, and reporting systems that help clinicians better understand when students might be struggling.

In addition to these health-related works at the main conference, several Microsoft authors are also contributing to the Symposium on Computing and Mental Health, which brings together researchers from multiple disciplines who seek to use digital solutions to help those in mental distress. One such work, Machine learning for Precise Targeting of a Mobile Dialectical Behavior Therapy Skills Training Application by Mary Czerwinski, Ran Gilad-Bachrach, Daniel McDuff, Kael Rowan, and Ann Paradiso, develops and evaluates a mobile application prototype that introduces skills related to dialectical behavior therapy (DBT), a common treatment for suicidal tendencies and borderline personality disorder. This work weaves together interactions with a chat-bot (“eMarsha”) with videos of Dr. Marsha Linehan, the creator of DBT, discussing therapeutic concepts and skills.  With this work, the authors explore new ground in using artificial intelligence and language understanding technologies to extend the reaches of the therapist.

The eyes have it: advancements in human gaze tracking

Just as eye contact is an important social-input signal in person-to-person interactions, for many years, computer science researchers have been training computers to detect where our eyes are focusing on to create new applications and experiences where humans and computers can work better in unison. Microsoft Research has made incredible progress during the last few years in advancing fundamental technologies that can accurately detect the focal-point of human gaze.

Looking Coordinated: Bidirectional Gaze Mechanisms for Collaborative Interaction with Virtual Characters, by Sean Andrist, Michael Gleicher, and Bilge Mutlu, received an honorable mention at CHI this year. They report on the design and evaluation of technologies for sharing social gaze cues in human-to-agent interactions in virtual reality (VR). Their work on bidirectional gaze mechanisms—the coordinated production and detection of social gaze cues—is based on a model synthesizing data collected in several human-to-human interactions.

These nonverbal cues can also help us coordinate with one another remotely. Researchers at Microsoft recently developed eye tracking technology to help remotely situated software engineers work together. In their paper: Improving Communication Between Pair Programmers Using Shared Gaze Awareness, Sarah D’Angelo and Andrew Begel designed a new visualization tool that allowed a pair of software engineers to see where in the code their partner was looking. In a study of the technology, remotely situated programmers working together to edit a program could talk more efficiently about the changes they wanted, and reported that knowing where their partner was looking made communicating easier.

Gaze tracking has also opened new possibilities for accessibility research. This year at CHI, Microsoft is presenting several projects related to accessibility, including a series of four papers (see below) around improving eye-gaze operated augmentative and alternative communications (AAC) technologies for people with severe motor and speech disabilities. These papers explore issues such as how to adapt interfaces to noise inherent in gaze input, how to dynamically adjust dwell-based eye typing algorithms to reduce text-entry errors, and how to augment AAC devices with secondary “awareness displays” to add backchannel content that enhances the richness of communication through low-bandwidth technologies.

In one of these works, the authors explore how to use smartphone cameras for simple eye-gesture-based text input. To mitigate the drawbacks of current approaches, we created Simple Smartphone Eyegaze Communication, an eye gesture communication system that runs on a smartphone, and is designed to be low cost, robust, portable, and easy to learn, with a high communication bandwidth. GazeSpeak can interpret eye gestures in real time, decode these gestures into predicted utterances, and facilitate communication, with different user interfaces for speakers and interpreters.

Papers:

  1. Toward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design” by Anna Maria Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish Kulkarni, Shaun Kane, Meredith Morris. Honorable Mention.
  2. Improving Dwell-Based Gaze Typing with Dynamic, Cascading Dwell Times” by Martez E Mott, Shane Williams, Jacob O. Wobbrock, Meredith R. Morris.
  3. Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities” by Xiaoyi Zhang, Harish Kulkarni, Meredith Ringel Morris.
  4. Exploring the Design Space of AAC Awareness Displays” by Kiley Sobel, Alexander Fiannaca, Jon Campbell, Harish Kulkarni, Ann Paradiso, Ed Cutrell, Meredith Morris. Honorable Mention.

These areas, along with the post last week about “Interacting through touch”, are just a sample of what we’re up to in HCI at Microsoft. For those interested in learning more about the full range of HCI projects we’re working on, our group page is a great starting point.

Related:

The post CHI Part 2: Creating technologies that enhance lives in the areas of health and eye tracking appeared first on Microsoft Research.

]]>
CHI Part 1: Interacting through touch, virtually and physically https://www.microsoft.com/en-us/research/blog/chi-1-interacting-touch-virtually-physically/ Thu, 04 May 2017 16:00:17 +0000 https://www.microsoft.com/en-us/research/?p=380660 A preview of the ACM CHI Conference on Human Factors in Computing Systems By Justin Cranshaw, Researcher, Microsoft Research Human-computer interaction (HCI) research at Microsoft provides a human-centered perspective on the future of computing, reshaping the roles computers play in our lives and propelling society forward through the tools, systems, and methods we investigate and […]

The post CHI Part 1: Interacting through touch, virtually and physically appeared first on Microsoft Research.

]]>
A preview of the ACM CHI Conference on Human Factors in Computing Systems

By Justin Cranshaw, Researcher, Microsoft Research

CHI Part 1: Interacting through touch, virtually and physically

Human-computer interaction (HCI) research at Microsoft provides a human-centered perspective on the future of computing, reshaping the roles computers play in our lives and propelling society forward through the tools, systems, and methods we investigate and invent. Microsoft Research’s HCI team has expertise on a wide range of domain-specialties and disciplines, providing guidance to much of the work we do at Microsoft, in both product and research.

One of the premier international venues for sharing academic work in HCI is the ACM CHI Conference on Human Factors in Computing Systems (or CHI, as it’s more commonly called). This year’s conference will be held May 6–11 in Denver, where 73 of us at Microsoft contributed to 37 papers, with 8 papers receiving awards.  In this post and the next, I’ll share some of the major themes I see in the work my Microsoft colleagues are presenting at CHI this year.

For decades, “touch” has been a major center of focus for HCI research to build more natural methods of computational input. This year, CHI features some of the latest Microsoft advances in touch-related research, including work on multi-touch sensing digital pen input that mirrors how people naturally use their hands. For example, in the physical world, when people hold (“touch”) a document, they will often frame a particular region of interest between thumb and forefinger in their non-preferred hand, while they mark up the page with a pen in their preferred hand.

One paper, As We May Ink? Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences, by Yann Riche, Nathalie Henry Riche, Ken Hinckley, Sarah Fuelling, Sarah Williams, and Sheri Panabaker, analyzes how people use freeform ink—on both “analog” paper and “digital” computer screens—in their everyday lives. This revealed a key insight, namely that ink is laden with many latent expressive attributes (such as the placement, size, and orientation of notations) that far exceed any textual transcription of the handwriting alone. In this and many other ways, ink uniquely affords the capture of fleeting thoughts, weaving together symbolic and figurative content, in a way that one could never achieve with a keyboard.

A second paper, WritLarge: Ink Unleashed by Unified Scope, Action, & Zoom, by Haijun Xia, Ken Hinckley, Michel Pahud, Xiao Tu, and William A.S. Buxton, earning an Honorable Mention award, introduces the system that puts these fundamental insights into action—on the mammoth 84-inch Microsoft Surface Hub, no less—to unleash the latent expressive power of ink. Using multitouch, the user can simply frame a portion of their “whiteboard” session between thumb and forefinger, and then act on such a selection (such as by copying, sharing, organizing, or otherwise transforming the content) using the pen wielded by the opposite hand.

And yet a third paper, Thumb + Pen Interaction on Tablets, by Ken Pfeuffer, Ken Hinckley, Michel Pahud, and William A.S. Buxton, addresses this notion of “pen + touch” interaction in laptop scenarios, such as when using a Surface tablet on the couch, where the nonpreferred hand must often hold the device itself. In this case, the thumb is available and sufficiently mobile to manipulate many controls, enabling a whole new space of “thumb + pen” interactions, such as allowing the user to readily shift use of a pen between annotation and cell-selection functions on an Excel spreadsheet.

HCI research on “touch” is not just about enabling new ways to interact with the virtual world through physical devices. Simulating the physicality of touch is also an important dimension for driving natural interactions in virtual worlds. Since the beginning of virtual reality (VR) development, the goal has been to immerse the users in a virtual world that will be as real to them as the physical one. While the capabilities of computer graphics and virtual reality display headsets, as well as 3D audio rendering, has progressed significantly during the last decade, our ability to simulate physicality in the virtual world is still in its infancy.

To help crack this problem, a team of Microsoft Research researchers—Eyal Ofek, Christian Holz, Hrvoje Benko, and Andy Wilson, along with Lung-Pan Chen, a visiting intern from HPI, Germany—has proposed a solution that can help us feel our environment while in VR. The authors built a system that tricks your senses by redirecting your hands, such that whenever you see yourself touching a virtual object, your hands are actually touching a physical geometry. Such a technology has the potential to make virtual worlds come alive in a tactile way.

Since Douglas Engelbart introduced the computer mouse in 1968, touch has been one of the major areas where HCI has advanced computation. Now, nearly 50 years later, Microsoft initiatives in HCI are helping reinvent our relationships with computation through new touch-based interactions and experiences.

Related:

The post CHI Part 1: Interacting through touch, virtually and physically appeared first on Microsoft Research.

]]>
Deep Learning for Program Synthesis https://www.microsoft.com/en-us/research/blog/deep-learning-program-synthesis/ Fri, 21 Apr 2017 16:00:53 +0000 https://www.microsoft.com/en-us/research/?p=378116 By Rishabh Singh, Jacob Devlin, Abdelrahman Mohamed, and Pushmeet Kohli, Microsoft Research Despite the many advances in computing over the past decades, the actual process of writing computer software has not fundamentally changed — a programmer must manually code the exact algorithmic logic of a program in a step-by-step manner using a specialized programming language. Although programming languages […]

The post Deep Learning for Program Synthesis appeared first on Microsoft Research.

]]>
By Rishabh Singh, Jacob Devlin, Abdelrahman Mohamed, and Pushmeet Kohli, Microsoft Research

Despite the many advances in computing over the past decades, the actual process of writing computer software has not fundamentally changed — a programmer must manually code the exact algorithmic logic of a program in a step-by-step manner using a specialized programming language. Although programming languages have become much more user-friendly over the years, learning how to program is still a major endeavor that most computer users have not undertaken.

In a recent paper, we report our latest work in deep learning for program synthesis, where deep neural networks learn how to generate computer programs based on a user’s intent. The user simply provides a few input/output (I/O) examples to specify the desired program behavior, and the system uses these to generate a corresponding program.

For example, imagine that a user has a list of names that she wishes to format in a specific way, as shown below. She provides a few input-output examples, and the system auto-fills the remaining outputs (shown in light gray). In cases where users have hundreds or thousands of input strings, this can be a major time saver.

input/output string for Program Synthesis

The system performs this task by generating a program in a domain specific language (DSL). The user does not have to understand the details of the DSL, and in fact she generally does not see the program at all. In our DSL, the correct program corresponding to the above example is:

Concat(
   ToCase(
       GetToken(
           input,
           Type=Word,
           Index=-1),
       Type=Proper),
   Const(", "),
   ToCase(
       SubString(
         GetToken(
             input,
             Type=Word,
             Index=1),
         Start=0,
         End=1),
       Type=Proper),
   Const("."))

There are two key challenges in program synthesis. First, there are trillions of possible programs in our expressive DSL, and the correct program has likely never been seen before by the system. Second, because the I/O examples are hand-generated by a human, they often contain noise (e.g., typos), as in the example above (Notice Useato instead of Uesato in the second example output).

Previous approaches to solving this problem — most notably the FlashFill system in Microsoft Excel — have relied on hand-crafted rules and heuristics to guide the search for the desired program. However, this approach makes it very difficult to extend the capabilities of the DSL, requires years of manual effort, and is also extremely sensitive to any noise in the I/O examples.

Our system, called RobustFill, leverages recent advances in deep learning to take a data-driven approach to program synthesis without the need for any hand-crafted rules. Instead, it uses an attentional sequence-to-sequence neural network, first pioneered for use in language translation, to generate the program based on the I/O examples. A sketch of our network is shown below, and the details are provided.

RobustFill for Program Synthesis

We trained our system on millions of randomly generated I/O + program pairs, but determined it could perform well on real-world data, because it could learn the semantics of the DSL. Overall, our system achieved 92 percent accuracy on a set of real-world benchmarks.  What’s especially encouraging is the system is able to maintain its high accuracy even when the I/O examples contain significant noise.

Implications for Programming

The ability to successfully train neural architectures for learning to program in a rich functional language (FlashFill DSL) not only marks an exciting breakthrough in neural program synthesis, but also is a small but noteworthy step towards achieving more general artificial intelligence. It addresses the key challenges of incorporating interpretability and also touches on the important topic of connecting distributed representations with symbolic representations of knowledge.

We are now working on extensions of these architectures for learning programs in DSLs with stateful variables and control flow to generate richer program classes.  We believe work along with direction will require us to study and tackle the fundamental technical issues involved in program synthesis and induction.

More information can be found in the papers below:

RobustFill: Neural Program Learning under Noisy I/O” by Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdelrahman Mohamed, and Pushmeet Kohli, in arXiv:1703.07469

Neuro-Symbolic Program Synthesis” by Emilio Parisotto, Abdelrahman Mohamed, Rishabh Singh, Lihong Li, Denny Zhou, and Pushmeet Kohli, in 5th International Conference on Learning Representations (ICLR 2017)

Contributing interns: Emilio Parisotto (CMU), Jonathan Uesato (MIT), Surya Bhuptairaju (MIT)

The post Deep Learning for Program Synthesis appeared first on Microsoft Research.

]]>
From cancer to crop genomics — using Research as a Service at the intersection of computers and biology https://www.microsoft.com/en-us/research/blog/cancer-crop-genomics-using-research-service/ Tue, 28 Mar 2017 13:00:49 +0000 https://www.microsoft.com/en-us/research/?p=373631 By Kenji Takeda, Director, Azure for Research, AI and Research Ever since Nicola Bonzanni was a little boy playing in the tiny Italian village of Bonate Sotto, just north of Milan, he was fascinated by nature and by building things. As he grew up, he wondered how computing and nature might be intertwined. “While I […]

The post From cancer to crop genomics — using Research as a Service at the intersection of computers and biology appeared first on Microsoft Research.

]]>
By Kenji Takeda, Director, Azure for Research, AI and Research

Nicola Bonzanni, Co-Founder of ENPICOM

Nicola Bonzanni, Co-Founder, ENPICOM

Ever since Nicola Bonzanni was a little boy playing in the tiny Italian village of Bonate Sotto, just north of Milan, he was fascinated by nature and by building things. As he grew up, he wondered how computing and nature might be intertwined. “While I was studying computer science I was inspired by nature in building new algorithms. But then I thought perhaps you could do it the other way around, basically taking things that are being developed in computer science and apply them to biological problems.”

After finishing his master’s thesis, he looked for opportunities to pursue his dream. He met a Dutch researcher while at a workshop in the south of France. The team at Vrije Universiteit Amsterdam was looking for a PhD student, so they interviewed Nicola. Despite his fears about his English proficiency, his passion for combining computer science and biology came across. Professor Jakob (Jaap) Heringa was impressed. “The fact that he already seemed to be able to combine computer science and life sciences was a very good sign to us, and we decided to hire him based on that.” So Nicola packed his bags and moved to Amsterdam, even though he had never been there before.

The next stop on Nicola’s path was developing novel methods for analyzing sequences of genes to look at how they evolve through a process known as phylogenetic fingerprinting. He teamed up with Professor Berthold Göttgens at the University of Cambridge to apply computer science to better understand leukemia, particularly how different genes in blood cells interact with each other.

Nicola wanted to share his work and was inspired to build a web-based tool called ConBind.org to help others around the world studying the same thing, delivering it as Research as a Service (RaaS). He chose this model of delivery so that researchers without access to powerful computers could still do the computationally intensive work. The idea for deploying his bioinformatics service in the cloud came to Nicola after he attended a Microsoft Azure for Research training session in Amsterdam and successfully applied for an award for his project. The cloud provides the on-demand scalability needed for running his complex bioinformatic queries, and he’s now using the Azure Batch service to make it even easier.

While his research originally focused on understanding cancer, the techniques are equally applicable to all living things: viruses, bacteria, plants and animals. That led Nicola to see if his tool could help farmers achieve such benefits as creating drought-resistant plants, or increasing crop yields. So, he contacted KeyGene, one of the world’s leading AgBioTech companies.

“KeyGene’s passion is crop innovation, getting new and improved crops or actually making the technology that we provide to plant-breeding companies and food industry,” explained Jan van Oeveren, program scientist lead discovery at KeyGene. Genomics is being used to identify key traits that are of interest to the plant breeder, for example disease resistance or shelf life. “We’re excited to work with Nicola to see if the ConBind tool can be applied to plant genomics. Currently we’re only looking at genes themselves. With ConBind, we’re also looking outside the genes to see how other parts of the genome of the DNA affect a certain trait.”

Nicola isn’t stopping there. His next project, in collaboration with a friend, Alvise Trevisan, is a startup called ENPICOM, supported by the Microsoft BizSpark program. This company combines their expertise in software engineering and life sciences to build the “Internet of Living Things.” This project aims to connect individuals with their environment, their health and their genomics, using Azure IoT Suite. Said Nicola, “The next step in the Internet of Living Things is using AI and Cortana Intelligence to make complex decisions using a multitude of different data sources.”

It’s been an amazing journey for Nicola, starting in Bonate Sotto, where his passion for nature and technology was born, to his exciting new startup ENPICOM. “I think that the challenge of this century is transforming life science in an engineering process and to do that you need to bring engineering into life science. And although probably I will not see the end of this journey, I would like to participate in it.”

Learn more

The post From cancer to crop genomics — using Research as a Service at the intersection of computers and biology appeared first on Microsoft Research.

]]>
Microsoft challenge: Build a collaborative AI in Minecraft https://www.microsoft.com/en-us/research/blog/microsoft-challenge-minecraft-collaborative-ai/ Tue, 14 Mar 2017 14:00:05 +0000 https://www.microsoft.com/en-us/research/?p=367736 By John Roach, Writer, Microsoft Research A Microsoft Research team has created a competition using Project Malmo, a platform that uses Minecraft as a testing ground for advanced artificial intelligence research. The team is challenging PhD students to develop an AI that learns to collaborate with other randomly assigned players to achieve a high score […]

The post Microsoft challenge: Build a collaborative AI in Minecraft appeared first on Microsoft Research.

]]>
By John Roach, Writer, Microsoft Research

Malmo Collaborative AI Challenge

A Microsoft Research team has created a competition using Project Malmo, a platform that uses Minecraft as a testing ground for advanced artificial intelligence research. The team is challenging PhD students to develop an AI that learns to collaborate with other randomly assigned players to achieve a high score in a mini-game within the virtual world.

The field of collaborative AI research involves the development of technologies that work with, empower and augment human capabilities. The ability to collaborate is key to the development of a general AI that can mimic the nuanced and complex way humans learn and make decisions. Such an AI would represent an evolution from task-specific AI technologies that recognize speech, translate languages and caption images.

That would allow researchers to develop technology that can comprehend the intent of others, develop a shared problem-solving strategy and coordinate activity to efficiently accomplish a common task. While these problems remain unsolved, recent progress in AI research provides a foundation to begin tackling them, notes the Microsoft team hosting the Malmo Collaborative AI Challenge.

The challenge is open to PhD students worldwide. After registration, teams of one to three members are supplied a task that consists of one or more mini-games. The goal is to develop an AI solution that learns how to work with other, randomly assigned players to achieve a high score in the game.

Participants submit their solutions to GitHub, including a one-minute video that shows off the AI agent and summarizes what is interesting about their approach, all the participant’s code, and a Readme file that explains the selected approach, design decisions and instructions on how to run the code.

The solutions will be evaluated on originality, performance, code quality and GitHub stars, which are a measure of popularity. Three winning teams will receive Azure research grants of US $20,000. Members of three winning teams from the European Economic Area and Switzerland are also eligible to win spots at the Microsoft Research AI Summer School.

Related:

John Roach writes about Microsoft research and innovation. Follow him on Twitter.

The post Microsoft challenge: Build a collaborative AI in Minecraft appeared first on Microsoft Research.

]]>
Democratizing AI to improve citizen health https://www.microsoft.com/en-us/research/blog/democratizing-ai-improve-citizen-health/ Mon, 13 Mar 2017 16:00:35 +0000 https://www.microsoft.com/en-us/research/?p=370100 By Kenji Takeda, Director, Azure for Research Doctors make life-saving — and life-changing — decisions every day. But how do they know that they are making the best decisions? Can artificial intelligence (AI) help? “Before evidence-based medicine, decision-making in health care was heavily reliant on the expertise and knowledge of the health professional, usually a […]

The post Democratizing AI to improve citizen health appeared first on Microsoft Research.

]]>
By Kenji Takeda, Director, Azure for Research

Doctors make life-saving — and life-changing — decisions every day. But how do they know that they are making the best decisions? Can artificial intelligence (AI) help?

“Before evidence-based medicine, decision-making in health care was heavily reliant on the expertise and knowledge of the health professional, usually a doctor. What has happened in the last 20, 30 years is that the health care literature has exploded way beyond what any individual doctor could hope to keep up with,” explained David Tovey, editor-in-chief at Cochrane. Cochrane is a not-for-profit organization that creates, publishes and maintains systematic reviews of health care interventions, with more than 37,000 contributors working in 130 countries.

The goal is to analyze the latest medical research to find the best treatments and interventions for patients and citizens. Systematic reviews bring together the best available research evidence from individual clinical trials and study data from around the world to inform the development of guidelines, individual practice decisions and national-scale health policymaking. These are rigorous and robust studies needed to provide the best advice, but they can take more than a year or two to complete. Tovey is concerned that systematic reviews, which are a painstaking process, are not used as widely as necessary. Said Tovey, “This is one of the main limitations for using systematic reviews to guide decision-making, particularly in policy areas. Policymakers increasingly want their decisions to be made in a period of weeks or months.”

Professor James Thomas at University College London (UCL) first started exploring text mining a decade ago because it promised to automate the laborious process of identifying and classifying medical research reports. In 2006, the state of the art in text mining was to develop complex rules to try to analyze these reports in an intelligent way.

But these rules-based approaches failed.

The Cochrane Transform Project is now applying AI and machine learning to analyze thousands of reports to automatically select ones to include in systematic reviews. This new approach is successfully saving weeks of monotonous work, freeing up the expert health care reviewers to spend their time and energy on high-level analysis. Thomas is using Cortana Intelligence to quickly develop and deploy AI solutions in the cloud. He explained, “What makes this particularly efficient is the fact that I can build a classifier using the studio and then just deploy it as a web service with the click of a button, without deploying a server.” The flexibility and open nature of Azure Machine Learning also means his team can be flexible in their approach. Said Thomas, “Initially I developed these models in R, which didn’t work as well as Python. So I was able to keep the top and the bottom of the process and just swap out the machine learning section in the middle.”

The team is continuously improving the performance of the machine learning system, and engaging with more than 5,000 citizen scientists in 117 countries through the Cochrane Crowd platform. “Cochrane Crowd has really enabled anybody who wants to get involved with producing health evidence to do so,” said Anna Noel-Storr of Cochrane’s Dementia Group. The Cochrane Crowd participants have screened more than 300,000 reports and 30,000 clinical trials to create a dataset to help train James’ AI. The machine learning system has improved the efficiency of the crowd by more than 60 percent. Thomas is excited by how this benefits everyone. “So what we end up with is a virtuous circle whereby the AI makes the crowd more efficient, and then the crowd, by supplying more data, makes the AI more accurate.”

These services are deployed through cloud APIs to help clinical assessment groups look at the latest medical research in different specialties, from cardiology and dementia, to public health issues such as obesity, healthy eating and physical exercise. The flexibility of the Microsoft Cloud means that the researchers can quickly create customized systems tailored for each clinical assessment group. This is then made usable by researchers worldwide through the EPPI-Reviewer cloud service, which is now running on Microsoft Azure. The AI system augments reviewers’ capabilities, allowing them to focus on understanding data instead of the monotonous work of sifting through thousands of research studies by hand.

The system is now being used by the National Institute for Health and Care Excellence in the U.K., which provides health care delivery guidance across the U.K. National Health Service (NHS) for more than 65 million people. It will directly speed up adoption of new health care interventions at national and international scale, for example, by facilitating the clearing of new drugs for widespread use across the NHS. The result is that decisions about which interventions are best will be faster and more reliable, thanks to the intelligent cloud.

Noel-Storr is excited by the prospect of democratizing AI for health, and how it can augment what people can achieve. “I think what excites me most about this work is that it is about exploring where those boundaries lie between what the machine can do and what the human can do…. so that we can then better direct human effort where it’s most needed.”

Learn more

The post Democratizing AI to improve citizen health appeared first on Microsoft Research.

]]>
A new understanding of the world through grassroots Data Science education at UC Berkeley https://www.microsoft.com/en-us/research/blog/grassroots-data-science-education-uc-berkeley/ Thu, 09 Mar 2017 17:00:03 +0000 https://www.microsoft.com/en-us/research/?p=368999 By Vani Mandava, Director, Data Science, Microsoft Research While some may regard data science as an easy passport to a job for the tech savvy, Luis Macias has different ideas. The fourth-year undergraduate student, who is majoring in American Studies at University of California, Berkeley (UC Berkeley), wants to turn the hype of data science […]

The post A new understanding of the world through grassroots Data Science education at UC Berkeley appeared first on Microsoft Research.

]]>
By Vani Mandava, Director, Data Science, Microsoft Research

Data Science education at UC Berkeley

Students at UC Berkeley Foundations of Data Science Program

While some may regard data science as an easy passport to a job for the tech savvy, Luis Macias has different ideas. The fourth-year undergraduate student, who is majoring in American Studies at University of California, Berkeley (UC Berkeley), wants to turn the hype of data science into hope for low-income communities like the one he grew up in.

Luis was among the first students to take UC Berkeley’s innovative Foundations of Data Science, an introductory data science course designed for freshman and sophomore students of all majors. The course is a key component of a multi-year university effort to forge a broader, more diverse, and inclusive scope for the emerging discipline of data science.

Recalling an assignment to gauge how water consumption data might relate to socioeconomic conditions, Luis explained how having the power to get and analyze data about his own ZIP Code ignited an understanding that data science can yield new insights capable of solving some of society’s most complex problems.

“The income level was around $25K, a number that was powerful for me in particular, because I think it explained a lot of the social issues and problems my community had,” he said.

Berkeley’s Data Science Education Program aims to make data science an integral feature of a liberal education and a core interdisciplinary capacity available to all Berkeley undergraduates. This is a bold experiment that will equip thousands of Berkeley students across campus with a fundamental education in data-driven thinking empowered by advanced statistical and computational techniques.

“The team of educators see their role as making it possible for students to bring to bear data science in all the ways they wish to use it in the world,” notes History professor Cathryn Carson. Carson is one of the faculty members leading the effort to build a diverse curriculum that includes advanced classes as well as connector courses that provide a bridge between familiar academic subjects and newly available data science techniques. “The energy and enthusiasm of students in the courses clearly demonstrate that the initiative will put data science to work in a breadth of domains that serve society, and UC Berkeley will play a particularly powerful role as a public university in this new data rich era.”

This year, the program is enabling more than a thousand students across 56 different undergraduate majors to learn critical computational and analytical skills demanded by the projected half million jobs in data science by 2018. That’s a lot of potentially unfilled jobs, an opportunity highlighted in numerous media accounts over the past couple of years.  In 2015, Forbes wrote about the urgent need for qualified data science workers who bring different skills, expertise, and experiences to the discipline. Some of them will no doubt emerge from Berkeley’s unique program of connector courses that represent students from a diverse range of skills and disciplines.

To succeed, the program had to be accessible to students beyond the realm of computer science. One way the program does this is through a flexible and scalable technology infrastructure that enables students to quickly set up labs for hands-on practice—they don’t have to spend time installing programs or learning nuances of complicated applications.

“By hosting it in Azure, we can control the environment,” said Ryan Lovett, Systems Manager for the Department of Statistics at UC Berkeley. “Students just log in and they’re ready to go.”

David Culler, professor of Electrical Engineering and Computer Sciences at UC Berkeley, believes the program can extend computational thinking to benefit more disciplines. He anticipates the program will equip students with the ability to extract their own insights from the world’s information and build tools that benefit people in society. He likens the ability to understand an increasingly complex world to a new form of perception — combining mathematical thinking and the arts with computational tools for new forms of expression.

Such data science projects include classic and new problems like music genre classification, text analysis of famous literary works, identifying insights from bike sharing data in San Francisco, or analyzing jury selection in Alameda County.

Berkeley prides itself as a place where the world’s brightest minds explore, ask questions and improve the world. Thanks to the Data Science Education Program, thousands of Berkeley students are better critical thinkers.

Note: Microsoft partners closely with UC Berkeley in support of its Data Science Education Program. Since 2015, Microsoft Research, through its Azure for Research program provided $235K in Azure credits to enable the Foundations of Data Science course along with $260K in research credits and $5K in training credits. Microsoft also provided $75K in unrestricted gift funding towards UC Berkeley’s Data Program.

Learn more

The post A new understanding of the world through grassroots Data Science education at UC Berkeley appeared first on Microsoft Research.

]]>