About
I am a senior researcher in at Microsoft Research. My research interests include computer vision for human-computer interaction, Augmented Reality (AR)/Virtual Reality (VR), Haptics and interactive projection mapping.
I am on the editorial board of IEEE Computer Graphics and Applications, co-chaired the 19th ACM SIGSPATIAL 2011, and on program committee for several leading conferences. I Held a visiting lecture position at the school of computer science, Interdisciplinary Center, Herzelia, Israel.
I obtained my PhD at the Hebrew University of Jerusalem, and founded a couple of companies in the area of computer graphics, including the successful Photon-Paint for the Amiga computer, and managing the software R&D at 3DV Systems, developing the world first active real-time depth cameras, later bought by Microsoft in 2008.
During 2004-2005 I was a researcher at Microsoft Research Asia (MSRA), working on issues such as video completion, reconstruction of hair from images, and camera based interaction, following by…
Research Highlights
Augmented Reality/Virtual Reality
I am interesting in the way that AR & VR effect and being effected by the real environment, and ways in which AR &VR can influence new experiences, for example social interaction.
Haptics
While visual display and 3D audio are progressing at a fast pase, Haptics is still and mostly untapped sense. Many times, due to the severe limitations of haptic rendering capability, there is a need to manipulate our senses to generate a convincing AR & VR experience of touch.
Interactive Projection Mapping
The use of projector enable large area augmentation that can be shared by multiple users.
Camera Based Interaction
Cameras are cheep and powerful sensors. I am looking in ways that we can leverage of cameras capabilities to compensate for the physical limitations of existing displays and input techniques.
Computer Vision
Text detection, Analysis by Synthesis, 2D & 3D completion
Introducing TORC: A rigid haptic controller that renders elastic objects
Oxymorons of haptics might be the new normal Consumer virtual reality (VR) systems can immerse users in wonderous virtual worlds, rich in sights and sounds. However, how many times have you reached for an object in VR or augmented reality…
Haptic Controllers: How Microsoft is making virtual reality tangible
Researchers at Microsoft strive to advance perhaps one of the most challenging areas of research and development in virtual reality. Mike Sinclair showcases four haptic controllers and discusses their goal to realize and deliver truly immersive and convincing tactile experiences…
Uncanny Valley and the Sense of Touch
The notion of an “uncanny valley” is well known in robotics[1]. This is when increasing the realism of a robot’s human-like appearance can lead to feelings of unease—or even revulsion—as a robot becomes more and more (but never quite fully)…
Giant steps and liberating spaces – Virtual Reality is making cool moves
Cool innovations are happening in how virtual reality researchers are resolving natural locomotion challenges and how they relate to story space, as well as in liberating users from the small, object-free player settings of today, to allow them to safely…
Projects
Publications
Videos
Downloads
Other
Professional Service
Co-Chair
PC Member
- ISMAR
- CVPR
- ACM SIGSPATIAL
- Pacific Graphics
- ISMAR 2015
- ISS 2018
Reviewer
- ACM SIGGRAPH,
- ACM SIGGRAPH Asia,
- ACM SIGSPATIAL,
- CVPR,
- ICCV,
- Pacific Graphics,
- IEEE transcation on PAMI
- CHI
- UIST
- ISMAR
Awards
- Talpiot program member 1984
- Charles Clor Scholarship, 1992
- Star Developer Award, Microsoft, 2006
- Best paper, UIST 2009
- Best Paper, CHI 2013
- Best Video, CHI 2013
- Best Paper, CSCW 2016
- Honorable Mentioned Paper, CHI 2018
Patents / Publication
| PATENTS / PUBLICATION # | YEAR OF PUBLICATION / ISSUED | TITLE | CITATIONS |
|---|---|---|---|
| 9355268 | 2016 | Managing access by applications to perceptual information | |
| 9298345 | 2016 | Block View For Geographic Navigation | |
| 9171403 | 2016 | Contour Completion For Augmenting Surface Reconstructions. | |
| 9167290 | 2015 | City Scene Video Sharing On Digital Maps. | |
| 9146119 | 2015 | Scrubbing Variable Content Paths | |
| 9123159 | 2015 | Interactive Geo-Positioning Of Imagery | |
| 9110921 | 2015 | Map Editing with Little User Input | |
| 9032042 | 2015 | Audio Presentation Of Condensed Spatial Contextual Information. | |
| 8970691 | 2015 | Removal Of Rayleigh Scattering From Images | |
| 9008859 | 2015 | Navigation Instructions Using Low-Bandwidth Signaling | |
| 8971641 | 2015 | Spatial Image Index And Associated Updating Functionality. | |
| 8970691 | 2015 | Removal Of Rayleigh Scattering From Images. | |
| 2014/0267,228 | 2014 | Mapping augmented reality experience to various Environments | 0 |
| 8,831,380 | 2014 | Viewing media in the context of street-level images | 0 |
| 8,823,797 | 2014 | Simulated video with extra viewpoints and enhanced resolution for traffic cameras | 0 |
| 8,817,049 | 2014 | Automated fitting of interior maps to general maps | 0 |
| 8,773,424 | 2014 | User interfaces for interacting with top-down maps of reconstructed 3-D scences | 0 |
| 8,774,520 | 2014 | Geo-relevance for images | 0 |
| 8,769,396 | 2014 | Calibration and annotation of video content | 0 |
| 2014/0180,695 | 2014 | GENERATION OF CONVERSATION TO ACHIEVE A GOAL | 0 |
| 2014/0180,833 | 2014 | Path Queries | 0 |
| 2014/0165,037 | 2014 | REUSABLE APPLICATION USER EXPERIENCE | 0 |
| 2014/0129,394 | 2014 | VIRTUAL CLOSET FOR STORING AND ACCESSING VIRTUAL REPRESENTATIONS OF ITEMS | 0 |
| 2014/0098,107 | 2014 | Transitioning Between Top-Down Maps and Local Navigation of Reconstructed 3-D Scenes | 0 |
| 8,694,383 | 2014 | Path queries | 0 |
| 8,688,368 | 2014 | Image-based localization for addresses | 0 |
| 8,675,018 | 2014 | Electromechanical surface of rotational elements for motion compensation of a moving object | 0 |
| 2014/0051,510 | 2014 | IMMERSIVE DISPLAY WITH PERIPHERAL ILLUSIONS | 0 |
| 8,645,230 | 2014 | Virtual closet for storing and accessing virtual representations of items | 0 |
| 8,640,020 | 2014 | Adjustable and progressive mobile device street view | 0 |
| 8,633,942 | 2014 | View generation using interpolated values | 0 |
| 8,624,902 | 2014 | Transitioning between top-down maps and local navigation of reconstructed 3-D scenes | 0 |
| 8,611,643 | 2013 | Spatially registering user photographs | 0 |
| 8,581,900 | 2013 | Computing transitions between captured driving runs | 0 |
| 8,577,170 | 2013 | Shadow detection in a single image | 0 |
| 2013/0287,307 | 2013 | DATA DIFFERENCE GUIDED IMAGE CAPTURING | 0 |
| 8,570,325 | 2013 | Filter and surfacing virtual content in virtual worlds | 0 |
| 8,560,515 | 2013 | Automatic generation of markers based on social interaction | 0 |
| 8,550,909 | 2013 | Geographic data acquisition by user motivation | 0 |
| 8,548,725 | 2013 | Visual assessment of landmarks | 0 |
| 8,549,627 | 2013 | Detection of objectionable videos | 0 |
| 2013/0246,512 | 2013 | COGNITIVE AGENT | 0 |
| 2013/0229,434 | 2013 | VIEWING MEDIA IN THE CONTEXT OF STREET-LEVEL IMAGES | 0 |
| 2013/0212,138 | 2013 | CURVE REPRESENTATIONS | 0 |
| 8,503,794 | 2013 | Data difference guided image capturing | 0 |
| 8,494,566 | 2013 | Hybrid mobile phone geopositioning | 0 |
| 8,487,927 | 2013 | Validating user generated three-dimensional models | 0 |
| 8,488,040 | 2013 | Mobile and server-side computational photography | 0 |
| 2013/0173,653 | 2013 | PATH COMPOSITION FOR PLANNING | 0 |
| 2013/0156,220 | 2013 | SELECTIVE SPATIAL AUDIO COMMUNICATION | 0 |
| 8,463,289 | 2013 | Depersonalizing location traces | 0 |
| 2013/0142,384 | 2013 | ENHANCED NAVIGATION THROUGH MULTI-SENSOR POSITIONING | 0 |
| 8,453,060 | 2013 | Panoramic ring user interface | 0 |
| 8,447,136 | 2013 | Viewing media in the context of street-level images | 4 |
| 8,446,367 | 2013 | Camera-based multi-touch mouse | 0 |
| 8,447,105 | 2013 | Data driven interpolation using geodesic affinity | 0 |
| 8,442,716 | 2013 | Identifying physical locations of entities | 0 |
| 8,433,136 | 2013 | Tagging video using character recognition and propagation | 1 |
| 8,428,908 | 2013 | Cognitive agent | 0 |
| 2013/0071,016 | 2013 | SHADOW DETECTION IN A SINGLE IMAGE | 1 |
| 2013/0051,623 | 2013 | Geo-Relevance for Images | 2 |
| 2013/0050,472 | 2013 | REMOVAL OF RAYLEIGH SCATTERING FROM IMAGES | 0 |
| 2013/0009,954 | 2013 | SPATIALLY REGISTERING USER PHOTOGRAPHS | 0 |
| 2013/0006,517 | 2013 | PROVIDING ROUTES THROUGH INFORMATION COLLECTION AND RETRIEVAL | 3 |
| 2013/0006,953 | 2013 | SPATIALLY ORGANIZED IMAGE COLLECTIONS ON MOBILE DEVICES | 0 |
| 2012/0331,009 | 2012 | MAP EDITING WITH LITTLE USER INPUT | 0 |
| 2012/0331,093 | 2012 | AUDIO PRESENTATION OF CONDENSED SPATIAL CONTEXTUAL INFORMATION | 0 |
| 8,340,416 | 2012 | Techniques for robust color transfer | 0 |
| 2012/0322,458 | 2012 | DEPERSONALIZING LOCATION TRACES | 0 |
| 2012/0315,992 | 2012 | GEOGRAPHIC DATA ACQUISITION BY USER MOTIVATION | 2 |
| 8,326,048 | 2012 | Geo-relevance for images | 2 |
| 8,319,796 | 2012 | Generating a texture from multiple images | 0 |
| 8,311,287 | 2012 | Validation and correction of map data using oblique images | 0 |
| 2012/0274,642 | 2012 | AUTOMATED FITTING OF INTERIOR MAPS TO GENERAL MAPS | 0 |
| 8,301,996 | 2012 | Annotating images with instructions | 0 |
| 8,295,589 | 2012 | Spatially registering user photographs | 0 |
| 8,296,194 | 2012 | Method, medium, and system for ranking dishes at eating establishments | 0 |
| 2012/0262,552 | 2012 | City scene video sharing on digital maps | 0 |
| 2012/0239,513 | 2012 | VIRTUAL CLOSET FOR STORING AND ACCESSING VIRTUAL REPRESENTATIONS OF ITEMS | 1 |
| 2012/0233,198 | 2012 | CREATING VIRTUAL WORLDS FROM EXISTING DATA STRUCTURES | 0 |
| 8,266,132 | 2012 | Map aggregation | 0 |
| 2012/0207,386 | 2012 | Updating A Low Frame Rate Image Using A High Frame Rate Image Stream | 0 |
| 2012/0210,171 | 2012 | COGNITIVE AGENT | 0 |
| 8,237,791 | 2012 | Visualizing camera feeds on a map | 3 |
| 2012/0155,778 | 2012 | Spatial Image Index and Associated Updating Functionality | 2 |
| 8,204,338 | 2012 | Factoring repeated content within and among images | 0 |
| 8,200,037 | 2012 | Importance guided image transformation | 0 |
| 2012/0141,046 | 2012 | MAP WITH MEDIA ICONS | 1 |
| 8,195,430 | 2012 | Cognitive agent | 0 |
| 2012/0133,665 | 2012 | Generating A Texture From Multiple Images | 0 |
| 8,189,925 | 2012 | Geocoding by image matching | 0 |
| 8,184,911 | 2012 | Flexible matching with combinational similarity | 1 |
| 2012/0109,942 | 2012 | IDENTIFYING PHYSICAL LOCATIONS OF ENTITIES | 1 |
| 2012/0084,323 | 2012 | GEOGRAPHIC TEXT SEARCH USING IMAGE-MINED DATA | 7 |
| 2012/0072,302 | 2012 | Data-Driven Item Value Estimation | 0 |
| 2012/0062,748 | 2012 | VISUALIZING VIDEO WITHIN EXISTING STILL IMAGES | 0 |
| 8,125,493 | 2012 | Generating a texture from multiple images | 0 |
| 2012/0027,250 | 2012 | DATA DIFFERENCE GUIDED IMAGE CAPTURING | 0 |
| 8,103,445 | 2012 | Dynamic map rendering as a function of a user parameter | 17 |
| 2012/0016,586 | 2012 | VISUAL ASSESSMENT OF LANDMARKS | 0 |
| 8,098,245 | 2012 | Smart navigation for 3D maps | 0 |
| 2011/0317,914 | 2011 | TECHNIQUES FOR ROBUST COLOR TRANSFER | 0 |
| 8,085,990 | 2011 | Hybrid maps with embedded street-side images | 3 |
| 2011/0312,374 | 2011 | MOBILE AND SERVER-SIDE COMPUTATIONAL PHOTOGRAPHY | 1 |
| 2011/0313,779 | 2011 | AUGMENTATION AND CORRECTION OF LOCATION BASED DATA THROUGH USER FEEDBACK | 1 |
| 2011/0299,766 | 2011 | DATA DRIVEN INTERPOLATION USING GEODESIC AFFINITY | 1 |
| 2011/0298,801 | 2011 | VIEW GENERATION USING INTERPOLATED VALUES | 0 |
| 2011/0298,928 | 2011 | SIMULATED VIDEO WITH EXTRA VIEWPOINTS AND ENHANCED RESOLUTION FOR TRAFFIC CAMERAS | 0 |
| 2011/0302,527 | 2011 | ADJUSTABLE AND PROGRESSIVE MOBILE DEVICE STREET VIEW | 0 |
| 2011/0294,515 | 2011 | HYBRID MOBILE PHONE GEOPOSITIONING | 2 |
| 2011/0286,660 | 2011 | Spatially Registering User Photographs | 3 |
| 8,060,302 | 2011 | Visual assessment of landmarks | 2 |
| 8,059,888 | 2011 | Semi-automatic plane extrusion for 3D modeling | 1 |
| 2011/0221,664 | 2011 | VIEW NAVIGATION ON MOBILE DEVICE | 3 |
| 8,009,178 | 2011 | Augmenting images for panoramic display | 4 |
| 2011/0200,252 | 2011 | FLEXIBLE MATCHING WITH COMBINATIONAL SIMILARITY | 0 |
| 2011/0195,781 | 2011 | MULTI-TOUCH MOUSE IN GAMING APPLICATIONS | 1 |
| 2011/0187,716 | 2011 | USER INTERFACES FOR INTERACTING WITH TOP-DOWN MAPS OF RECONSTRUCTED 3-D SCENES | 2 |
| 2011/0187,704 | 2011 | GENERATING AND DISPLAYING TOP-DOWN MAPS OF RECONSTRUCTED 3-D SCENES | 0 |
| 2011/0187,723 | 2011 | TRANSITIONING BETWEEN TOP-DOWN MAPS AND LOCAL NAVIGATION OF RECONSTRUCTED 3-D SCENES | 0 |
| 7,991,283 | 2011 | Geotagging photographs using annotations | 4 |
| 2011/0173,565 | 2011 | VIEWING MEDIA IN THE CONTEXT OF STREET-LEVEL IMAGES | 4 |
| 7,967,451 | 2011 | Multi-directional image displaying device | 7 |
| 2011/0137,741 | 2011 | PATH QUERIES | 0 |
| 7,957,596 | 2011 | Flexible matching with combinational similarity | 4 |
| 2011/0130,950 | 2011 | TRAVEL DIRECTIONS WITH TRAVEL-TIME ESTIMATES | 5 |
| 7,946,921 | 2011 | Camera based orientation for mobile devices | 1 |
| 7,941,271 | 2011 | Displaying images related to a requested path | 4 |
| 7,912,637 | 2011 | Landmark-based routing | 8 |
| 7,886,978 | 2011 | Techniques for decoding images of barcodes | 5 |
| 7,889,947 | 2011 | Image completion | 1 |
| 2011/0018,902 | 2011 | HYBRID MAPS WITH EMBEDDED STREET-SIDE IMAGES | 2 |
| 2010/0325,589 | 2010 | BLOCK VIEW FOR GEOGRAPHIC NAVIGATION | 4 |
| 2010/0316,300 | 2010 | DETECTION OF OBJECTIONABLE VIDEOS | 2 |
| 2010/0315,411 | 2010 | COMPUTING TRANSITIONS BETWEEN CAPTURED DRIVING RUNS | 1 |
| 7,852,315 | 2010 | Camera and acceleration based interface for presentations | 2 |
| 2010/0312,609 | 2010 | Personalizing Selection of Advertisements Utilizing Digital Image Analysis | 5 |
| 2010/0312,470 | 2010 | Scrubbing Variable Content Paths | 0 |
| 2010/0313,113 | 2010 | Calibration and Annotation of Video Content | 0 |
| 2010/0310,182 | 2010 | GEOCODING BY IMAGE MATCHING | 0 |
| 7,840,032 | 2010 | Street-side maps and paths | 7 |
| 7,831,089 | 2010 | Modeling and texturing digital surface models in a mapping application | 2 |
| 2010/0265,178 | 2010 | CAMERA-BASED MULTI-TOUCH MOUSE | 7 |
| 2010/0246,965 | 2010 | TAGGING VIDEO USING CHARACTER RECOGNITION AND PROPAGATION | 6 |
| 2010/0250,120 | 2010 | MANAGING STORAGE AND DELIVERY OF NAVIGATION IMAGES | 5 |
| 2010/0246,890 | 2010 | DETECTION OF OBJECTS IN IMAGES | 2 |
| 2010/0251,169 | 2010 | AUTOMATIC GENERATION OF MARKERS BASED ON SOCIAL INTERACTION | 2 |
| 2010/0245,361 | 2010 | CONTEXT-BASED MANAGEMENT OF MARKERS | 1 |
| 2010/0250,126 | 2010 | VISUAL ASSESSMENT OF LANDMARKS | 1 |
| 2010/0250,196 | 2010 | COGNITIVE AGENT | 1 |
| 2010/0245,344 | 2010 | ANNOTATING OR EDITING THREE DIMENSIONAL SPACE | 0 |
| 2010/0245,376 | 2010 | FILTER AND SURFACING VIRTUAL CONTENT IN VIRTUAL WORLDS | 0 |
| 2010/0250,366 | 2010 | MERGE REAL-WORLD AND VIRTUAL MARKERS | 0 |
| 7,805,066 | 2010 | System for guided photography based on image capturing device rendered user recommendations according to embodiments | 2 |
| 2010/0241,525 | 2010 | IMMERSIVE VIRTUAL COMMERCE | 3 |
| 2010/0241,514 | 2010 | AUTOMATED PROVISION OF USER FEEDBACK | 1 |
| 2010/0241,946 | 2010 | ANNOTATING IMAGES WITH INSTRUCTIONS | 0 |
| 2010/0235,076 | 2010 | ESTIMATION OF FUEL CONSUMPTION FROM GPS TRAILS | 0 |
| 2010/0235,078 | 2010 | DRIVING DIRECTIONS WITH MAPS AND VIDEOS | 0 |
| 2010/0235,356 | 2010 | ORGANIZATION OF SPATIAL SENSOR DATA | 0 |
| 2010/0225,665 | 2010 | MAP AGGREGATION | 0 |
| 2010/0218,094 | 2010 | SECOND-PERSON AVATARS | 0 |
| 7,777,648 | 2010 | Mode information displayed in a mapping application | 21 |
| 2010/0134,484 | 2010 | THREE DIMENSIONAL JOURNALING ENVIRONMENT | 2 |
| 2010/0118,025 | 2010 | MODE INFORMATION DISPLAYED IN A MAPPING APPLICATION | 55 |
| 2010/0085,350 | 2010 | OBLIQUE DISPLAY WITH ADDITIONAL DETAIL | 4 |
| 2010/0085,371 | 2010 | OPTIMAL 2D TEXTURING FROM MULTIPLE IMAGES | 0 |
| 2010/0080,551 | 2010 | Geotagging Photographs Using Annotations | 1 |
| 2010/0080,466 | 2010 | Smart Navigation for 3D Maps | 0 |
| 2010/0080,489 | 2010 | Hybrid Interface for Interactively Registering Images to Digital Models | 0 |
| 2010/0074,538 | 2010 | VALIDATION AND CORRECTION OF MAP DATA USING OBLIQUE IMAGES | 1 |
| 2009/0323,029 | 2009 | MULTI-DIRECTIONAL IMAGE DISPLAYING DEVICE | 2 |
| 2009/0285,482 | 2009 | DETECTING TEXT USING STROKE WIDTH BASED TEXT DETECTION | 4 |
| 2009/0284,527 | 2009 | VALIDATING USER GENERATED THREE-DIMENSIONAL MODELS | 4 |
| 2009/0237,510 | 2009 | VISUALIZING CAMERA FEEDS ON A MAP | 8 |
| 2009/0208,110 | 2009 | FACTORING REPEATED CONTENT WITHIN AND AMONG IMAGES | 2 |
| 2009/0190,857 | 2009 | IMPORTANCE GUIDED IMAGE TRANSFORMATION | 2 |
| 2009/0162,042 | 2009 | GUIDED PHOTOGRAPHY BASED ON IMAGE CAPTURING DEVICE RENDERED USER RECOMMENDATIONS | 3 |
| 7,548,659 | 2009 | Video enhancement | 20 |
| 2009/0141,966 | 2009 | INTERACTIVE GEO-POSITIONING OF IMAGERY | 6 |
| 2009/0110,327 | 2009 | Semi-automatic plane extrusion for 3D modeling | 0 |
| 2009/0092,277 | 2009 | Geo-Relevance for Images | 4 |
| 2009/0078,772 | 2009 | Techniques for decoding images of barcodes | 7 |
| 2009/0079,752 | 2009 | Generating a texture from multiple images | 2 |
| 2009/0058,855 | 2009 | ELECTROMECHANICAL SURFACE OF ROTATIONAL ELEMENTS FOR MOTION COMPENSATION OF A MOVING OBJECT | 0 |
| 7,477,236 | 2009 | Remote control of on-screen interactions | 4 |
| 2009/0002,394 | 2009 | AUGMENTING IMAGES FOR PANORAMIC DISPLAY | 9 |
| 2009/0003,702 | 2009 | IMAGE COMPLETION | 3 |
| 2008/0319,659 | 2008 | LANDMARK-BASED ROUTING | 16 |
| 2008/0319,660 | 2008 | LANDMARK-BASED ROUTING | 11 |
| 2008/0319,658 | 2008 | LANDMARK-BASED ROUTING | 7 |
| 2008/0273,795 | 2008 | FLEXIBLE MATCHING WITH COMBINATIONAL SIMILARITY | 8 |
| 2008/0249,702 | 2008 | Image-based localization for addresses | 1 |
| 2008/0120,023 | 2008 | DISPLAYING IMAGES RELATED TO A REQUESTED PATH | 3 |
| 2008/0066,000 | 2008 | PANORAMIC RING USER INTERFACE | 2 |
| 2008/0050,011 | 2008 | MODELING AND TEXTURING DIGITAL SURFACE MODELS IN A MAPPING APPLICATION | 6 |
| 2008/0043,020 | 2008 | USER INTERFACE FOR VIEWING STREET SIDE IMAGERY | 29 |
| 2007/0273,558 | 2007 | DYNAMIC MAP RENDERING AS A FUNCTION OF A USER PARAMETER | 64 |
| 2007/0236,451 | 2007 | Camera and Acceleration Based Interface for Presentations | 3 |
| 2007/0210,937 | 2007 | DYNAMIC RENDERING OF MAP INFORMATION | 61 |
| 2007/0076,920 | 2007 | STREET SIDE MAPS AND PATHS | 27 |
| 2006/0294,538 | 2006 | Inserting advertising content into video programming | 14 |
| 2006/0287,083 | 2006 | Camera based orientation for mobile devices | 8 |
| 2006/0257,042 | 2006 | Video enhancement | 28 |
| 2006/0248,462 | 2006 | Remote control of on-screen interactions | 2 |
