{"id":704044,"date":"2020-11-19T12:28:57","date_gmt":"2020-11-19T20:28:57","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=704044"},"modified":"2022-11-01T08:59:18","modified_gmt":"2022-11-01T15:59:18","slug":"the-future-of-work-unbound-2020-and-the-strange-new-mobility-of-space-and-time","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/the-future-of-work-unbound-2020-and-the-strange-new-mobility-of-space-and-time\/","title":{"rendered":"The future of work, unbound: 2020 and the strange new mobility of space and time"},"content":{"rendered":"<aside id=accordion-22f6d16f-20e5-42f3-ac0d-e0b4eb49e806 class=\"msr-table-of-contents-block accordion mb-5 pb-0\" data-bi-aN=\"table-of-contents\">\n\t<button class=\"btn btn-collapse bg-gray-100 mb-0 display-flex justify-content-between\" type=\"button\" data-mount=\"collapse\" data-target=\"#accordion-collapse-22f6d16f-20e5-42f3-ac0d-e0b4eb49e806\" aria-expanded=\"true\" aria-controls=\"accordion-collapse-22f6d16f-20e5-42f3-ac0d-e0b4eb49e806\">\n\t\t<span class=\"msr-table-of-contents-block__label subtitle\">In this article<\/span>\n\t\t<span class=\"msr-table-of-contents-block__current mr-4 text-gray-600 font-weight-normal\" aria-hidden=\"true\"><\/span>\n\t<\/button>\n\t<div id=\"accordion-collapse-22f6d16f-20e5-42f3-ac0d-e0b4eb49e806\" class=\"msr-table-of-contents-block__collapse-wrapper collapse show\" data-parent=\"#accordion-22f6d16f-20e5-42f3-ac0d-e0b4eb49e806\">\n\t\t<div class=\"accordion-body bg-gray-100 border-top pt-4\">\n\t\t\t<ol class=\"msr-table-of-contents-block__list\">\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#surfacefleet-mobility-as-transitions-of-user-activity-from-one-place-to-another\" class=\"msr-table-of-contents-block__list-item-link\">SurfaceFleet: Mobility as transitions of user activity from one \u2018place\u2019 to another<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#ambrosia-programming-as-if-failure-doesnt-matter\" class=\"msr-table-of-contents-block__list-item-link\">Ambrosia: Programming as if failure doesn\u2019t matter<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#changes-in-display-orientation-as-a-nuanced-transition-in-place\" class=\"msr-table-of-contents-block__list-item-link\">Changes in display orientation as a nuanced transition in \u2018place\u2019<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#closing-thoughts\" class=\"msr-table-of-contents-block__list-item-link\">Closing thoughts<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t<\/div>\n\t<span class=\"msr-table-of-contents-block__progress-bar\"><\/span>\n<\/aside>\n\n\n\n<p>For those of us who have transitioned to working from home over the course of the last year, we must navigate a strange new manifestation of <em>mobility<\/em>.<\/p>\n\n\n\n<p>Far-flung colleagues appear almost magically in grid format on a screen right in front of our faces, despite their remote locations. Yet at the same time, a document, presentation, piece of content, or part of a running application already at our fingertips is awkward to share with others on the same video call.<\/p>\n\n\n\n<p>It\u2019s a paradoxical science fiction world where far is near, and the close-at-hand dilates impossibly beyond our reach. Perhaps these surreal distortions of time and space explain in part why \u201cvideo-conferencing syndrome\u201d feels so draining.<\/p>\n\n\n\n<p>And even as we\u2019re stranded within the confines of our improvised home offices, we\u2019re somehow supposed to navigate this otherworldly place\u2014a jumbled chaos terrain of home and work, personal and professional, private and semi-public.<\/p>\n\n\n\n<p>Moving between these realities, sometimes moment by moment, makes us nimble in a way we\u2019ve never experienced before: our activity is mobile even as we stay put in the same location. We work in the same physical <em>spaces<\/em>, but as we navigate these transitions, we\u2019re not in the same human <em>places<\/em>.<\/p>\n\n\n\n<p>While 2020 has accelerated this trend, perhaps it\u2019s inevitable\u2014and indeed, as we point out below, in many ways this strange new mobility has been a long time coming.<\/p>\n\n\n\n<p>Microsoft researchers are creating technologies to help people succeed in this new way of life. We are working to develop systems that help us to navigate these changes. A new world where these transitions feel less strange\u2014and more empowering. A <em>place <\/em>appropriate to our current task, locality, and context, where \u201cmobility\u201d means technology that rises to the universal human need to connect and work with others seamlessly.<\/p>\n\n\n\n<p>To that end, Microsoft researchers have published three papers\u2014two of which appear at this year\u2019s <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/uist-2020\/\">ACM Symposium on User Interface Software and Technology<\/a> (UIST 2020)\u2014on new technologies that redefine how we interpret this concept of place.<\/p>\n\n\n\n<p>The first explores SurfaceFleet, a system to decouple computing from individual devices and places. The second presents Ambrosia, a system which uses resilient distributed programming techniques to unbind running programs and their state from any particular device (CPU). The third circles back on this notion of place, showing how nuanced social cues such as tilt and orientation of a display, on the adjustable Microsoft Surface Studio, can support elegant and natural transitions between different tasks and ways of using such a display.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"1080\" style=\"aspect-ratio: 1920 \/ 1080;\" width=\"1920\" controls src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/Fleet-Tilt-1m.mp4\"><\/video><\/figure>\n\n\n\n<h2 id=\"surfacefleet-mobility-as-transitions-of-user-activity-from-one-place-to-another\">SurfaceFleet: Mobility as transitions of user activity from one \u2018place\u2019 to another<\/h2>\n\n\n\n<p>What the Fleet system is in brief:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>A distributed system leveraging a robust, performant declarative database foundation and building on the Ambrosia runtime<\/li><li>An exploration of novel implications for migration of user experiences across devices<\/li><li>A platform for Applets: lightweight, distributed user interface elements that unbind interactions from devices, applications, users, and time<\/li><li>A collaboration tool enabling people to work across devices and act at synchronous or asynchronous times.<\/li><\/ul>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<article class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<div class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">PUBLICATION <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/surfacefleet-exploring-distributed-interactions-unbounded-from-device-application-user-and-time-uist-2020-paper\/\" data-bi-cN=\"SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and Time\" data-external-link=\"false\" data-bi-aN=\"citation\" data-bi-type=\"annotated-link\" class=\"annotations__link font-weight-semibold text-decoration-none\"><span>SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and Time<\/span>&nbsp;<span class=\"glyph-in-link glyph-append glyph-append-chevron-right\" aria-hidden=\"true\"><\/span><\/a>\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">Frederik Brudy, David Ledo, Michel Pahud, Nathalie Henry Riche, Christian Holz, Anand Waghmare, Hemant Surale, Marcus Peinado, Xiaokuan Zhang, Shannon Joyner, Badrish Chandramouli, Umar Farooq Minhas, Jonathan Goldstein, Bill Buxton, Ken Hinckley. <\/p>\n\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n<p>The COVID-19 pandemic has brought the collision of home and work\u2014of activity in limited physical <em>spaces <\/em>that must transition between different human places\u2014to a critical juncture.<\/p>\n\n\n\n<p>But two trends, manifested over the past decade already, are influencing the future of human experiences with computing technology.<\/p>\n\n\n\n<p>The first trend concerns hardware and systems architecture. With Moore\u2019s Law at an end, yet networking and storage exhibiting exponential gains, the future appears to favor systems that emphasize seamless mobility of data\u2014favoring techniques that consume network and storage bandwidth rather than those using a particular CPU. Accelerated by pervasive cloud services and 5G, these computational shifts show no sign of slowing.<\/p>\n\n\n\n<p>The second trend is one of human behavior. People now interact with more devices than ever before. Modern information work increasingly relies on multi-device workflows and distributed workspaces, with connected and interdependent devices\u2014smartphones, desktops, tablets, and perhaps even emerging new form factors. The problem here lies in that transitioning from device to device, and more broadly place to place, can cost us precious efficiency or resources, like time or attention, leaving our activities marooned on islands of glass instead of creating a new interconnected world.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and Time\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/_MXixOADlxg?feature=oembed&rel=0\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>What\u2019s needed is an ecosystem of technologies that seamlessly transitions from place to place, whether that \u201cplace\u201d takes the form of a literal location, a different device form factor, the presence of a collaborator, or the availability of the pieces of information needed to complete a particular task at a given time. Such a \u201cSociety of Technologies\u201d favors techniques that establish meaningful relationships between the members of this society, rather than with any particular device, to afford mobility of user activity from one place to another, in a very general sense of the word.<\/p>\n\n\n\n<p>Through this lens, we can view the essence of mobility as the transition of user activity from one place to another.&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/surfacefleet\/\" target=\"_blank\" rel=\"noreferrer noopener\">SurfaceFleet<\/a>&nbsp;is a working system, development toolkit, and user experience that explores some implications of these challenges by decoupling computation\u2014including its representation in the graphical user interface\u2014from the current&nbsp;<em>device<\/em>.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Yet, once&nbsp;user interface mechanisms&nbsp;are decoupled&nbsp;from&nbsp;a single&nbsp;device,&nbsp;we discovered that&nbsp;this also has interesting carry-on implications for unbinding interaction from the current&nbsp;<em>application<\/em>, the current&nbsp;<em>user<\/em>, and the current&nbsp;<em>time<\/em>, as well.&nbsp;The Fleet&nbsp;system handles transitions in place\u2014bridging the resulting gaps\u2014across all four of these dimensions.&nbsp;See the embedded video above for demonstrations that show how user interfaces can \u201cfloat\u201d above the screen and transcend program state that is confined to the current device.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"589\" height=\"393\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/SurfaceFleet_highres_fig-1.jpg\" alt=\"Surface Fleet\" class=\"wp-image-704539\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/SurfaceFleet_highres_fig-1.jpg 589w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/SurfaceFleet_highres_fig-1-300x200.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/SurfaceFleet_highres_fig-1-16x12.jpg 16w\" sizes=\"auto, (max-width: 589px) 100vw, 589px\" \/><figcaption>The Fleet system unbinds UI elements from not only the device but also the current application, user, and time. In the visible UI, <em>Applets <\/em>unbind controls from applications. <em>Portfolios <\/em>unbind tools, inputs, behaviors, and content from the current device and user. <em>Promises <\/em>unbind actions from time. Read more in the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/surfacefleet-exploring-distributed-interactions-unbounded-from-device-application-user-and-time-uist-2020-paper\/\">SurfaceFleet paper<\/a>.<\/figcaption><\/figure>\n\n\n\n<p>But authoring distributed programs is difficult and requires considerable expertise. How do we reimagine this notion of \u201cdevice\u201d that is so deeply baked into current development practices? This is where our journey crosses paths with a new distributed-systems technology known as Ambrosia.<\/p>\n\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"1002645\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: AI-POWERED EXPERIENCE<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300 display-block\" href=\"https:\/\/aka.ms\/research-copilot\/?OCID=msr_researchforum_Copilot_MCR_Blog_Promo\" aria-label=\"Microsoft research copilot experience\" data-bi-cN=\"Microsoft research copilot experience\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/01\/MSR-Chat-Promo.png\" alt=\"\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">Microsoft research copilot experience<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p id=\"microsoft-research-copilot-experience\" class=\"large\">Discover more about research at Microsoft through our AI-powered experience<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/aka.ms\/research-copilot\/?OCID=msr_researchforum_Copilot_MCR_Blog_Promo\" aria-describedby=\"microsoft-research-copilot-experience\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" data-bi-cN=\"Microsoft research copilot experience\" target=\"_blank\">\n\t\t\t\t\t\t\tStart now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n\n<h2 id=\"ambrosia-programming-as-if-failure-doesnt-matter\">Ambrosia: Programming as if failure doesn\u2019t matter<\/h2>\n\n\n\n<p>What the Ambrosia runtime does in brief:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Introduces the notion of \u201cvirtual resiliency,\u201d which allows programmers of distributed applications to program as if failure doesn\u2019t matter<\/li><li>Facilitates recovery and replay of logged messages that include mechanisms to correctly handle non-determinism<\/li><li>Achieves highly performant remote procedure calls through database techniques such as batching, high-performance log writing, high-performance serialization concepts, and group commit strategies<\/li><li>Provides the technical foundation of the Fleet system<\/li><\/ul>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<article class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<div class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">PUBLICATION<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/a-m-b-r-o-s-i-a-providing-performant-virtual-resiliency-for-distributed-applications\/\" data-bi-cN=\"A.M.B.R.O.S.I.A: Providing Performant Virtual Resiliency for Distributed Applications\" data-external-link=\"false\" data-bi-aN=\"citation\" data-bi-type=\"annotated-link\" class=\"annotations__link font-weight-semibold text-decoration-none\"><span>A.M.B.R.O.S.I.A: Providing Performant Virtual Resiliency for Distributed Applications<\/span>&nbsp;<span class=\"glyph-in-link glyph-append glyph-append-chevron-right\" aria-hidden=\"true\"><\/span><\/a>\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">Jonathan Goldstein, Ahmed Abdelhamid\u1d7b, Mike Barnett, Sebastian Burckhardt, Badrish, Chandramouli, Darren Gehring, Niel Lebeck, Christopher Meiklejohn, Umar Farooq Minhas, Ryan Newton, Rahee Ghosh Peshawaria, Tal Zaccai, Irene Zhang\n<\/p>\n\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n<p>Programmers face complex decisions and coding tasks when coping with failure in distributed systems\u2014especially when applications modify state that is shared across devices. Unfortunately, a lot can go wrong even in simple scenarios of passing messages between distributed services. Connections can drop. Distributed clients can crash at any moment. A remote procedure call (RPC) might even fail just as it sends a remote message, creating uncertainty of what has been sent or received that must be reconciled. All of these cases and error conditions must be correctly anticipated, handled, and implemented correctly, and in an efficient manner. This is why distributed services are so hard to program and deploy correctly.<\/p>\n\n\n\n<p>But using Ambrosia, a developer can write the code for their client as if failure doesn\u2019t matter. We call this <em>virtual resiliency<\/em>, similar to virtual memory, where one can author programs as if limits on physical memory don\u2019t exist.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"709\" height=\"366\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-blog-fig-2.png\" alt=\"Figure\" class=\"wp-image-704098\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-blog-fig-2.png 709w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-blog-fig-2-300x155.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-blog-fig-2-16x8.png 16w\" sizes=\"auto, (max-width: 709px) 100vw, 709px\" \/><figcaption>Virtual resilience for \u201cAlice\u2019s\u201d running client, where the Ambrosia runtime intercepts all outgoing and incoming remote procedure calls and logs them to resilient storage, such as in the cloud, before Alice acts on them. Ambrosia automatically replays this log to recover from failures in a way that ensures deterministically ordered, one-time delivery of requests.<\/figcaption><\/figure>\n\n\n\n<p>The developer simply wraps their service (\u201cAlice\u201d in the figure above) in the Ambrosia runtime. Ambrosia intercepts each message and logs it to resilient storage before sending it over the network via RPC. Whenever a remote service (\u201cBob\u201d) responds, Ambrosia likewise logs these return messages before Alice\u2019s code acts on their contents.<\/p>\n\n\n\n<p>Ambrosia encapsulates the many possible failure conditions, factoring all the distributed-systems complexity of the resulting client code. If Alice goes down, Ambrosia automatically recovers by replaying the log, allowing Alice\u2019s code to pick up where it left off. Likewise, if Bob crashes, the system can automatically recover from that, too, so long as Bob is wrapped in the Ambrosia runtime, as well. And since network connection state is also logged to resilient storage in the cloud via Azure, we also can automatically self-heal disruptions such as intermittent connections or changing network addresses via a subsystem known as the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/cra-enabling-data-intensive-applications-in-containerized-environments\/\">Common Runtime for Applications (CRA)<\/a>.<\/p>\n\n\n\n<p>Programming distributed applications in this way, as if failure doesn\u2019t matter, is a nifty trick. But the true secret sauce of Ambrosia is that it provides this virtual resiliency with high performance. It does so by applying decades-old wisdom that has been used to build performant, reliable, and available database systems. For instance, Ambrosia makes extensive use of batching, high-performance log writing, high-performance serialization concepts, and group commit strategies. It also includes mechanisms to properly handle non-determinism by logging any such events, as well. These carefully implemented techniques allow Ambrosia to deterministically provide virtual resiliency with little or no reduction in throughput, depending on message size, as compared to popular RPC frameworks that lack resilience mechanisms.<\/p>\n\n\n\n<p>With a distributed system built on these abstractions, we end up with two coordinated instances of the Ambrosia runtime surrounding the running services Alice and Bob:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"974\" height=\"330\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-Figure-3.png\" alt=\"Figure\" class=\"wp-image-704113\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-Figure-3.png 974w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-Figure-3-300x102.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-Figure-3-768x260.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-Figure-3-16x5.png 16w\" sizes=\"auto, (max-width: 974px) 100vw, 974px\" \/><figcaption>The running services &#8220;Alice&#8221; and &#8220;Bob&#8221; are encapsulated in coordinated instances of the Ambrosia runtime to ensure mutual resiliency of the distributed system.<\/figcaption><\/figure>\n\n\n\n<p>This basic architecture not only encapsulates many types of distributed-system failures, but it also allows for interesting variations, such as standing up multiple active instances of a service (so-called active\/active configurations) so that we can quickly <em>failover<\/em> to \u201cBob 2\u201d or \u201cAlice 2\u201d if one of the services dies or is slow to recover.<\/p>\n\n\n\n<p>But such a failover might not reflect a networking or system failure at all.<\/p>\n\n\n\n<p>Perhaps it is a matter of choice\u2014the end user\u2019s preference.<\/p>\n\n\n\n<p>Maybe a user of the Alice service shuts off their desktop at the end of a hectic day and picks up their tablet instead. \u201cDesktop Alice\u201d halts, and \u201cTablet Alice\u201d resumes where they left off. Instead of a network or hardware crash, it\u2019s simply a failover to their preferred device.<\/p>\n\n\n\n<p>This leads us to a key insight. Circling back to the Fleet system, where we started, we can now cast transitions of user activity from one device to another as a special case of failover. In this case, it\u2019s a transition from one device to another.<\/p>\n\n\n\n<p>But migration of program state to a new device is just one special case of mobility. If we have the right feedback and user interface mechanisms in place, we can generalize this as transitions of user activity from one place to another, in many senses of the word <em>place<\/em>. How best to do this is still an open problem. Our work explores some possibilities and hints at solutions. This suggests that cross-device and distributed systems will have major impact on user interfaces going forward, even if the full vista of interactive systems and human experiences this makes possible has only just begun to dawn.<\/p>\n\n\n\n<p>Next, we take this up a level by looking at a simple example of how sensing shifts in context\u2014such as responding appropriately when a user tilts a display\u2014can drive lightweight and natural transitions from one human activity to another.<\/p>\n\n\n\n<h2 id=\"changes-in-display-orientation-as-a-nuanced-transition-in-place\">Changes in display orientation as a nuanced transition in \u2018place\u2019<\/h2>\n\n\n\n<p>What tilt-responsive techniques for digital drawing boards do in brief:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Run on a Microsoft Surface Studio 2 using a C# module for sampling the sensors and implementing signal conditioning and a JavaScript-based client<\/li><li>Demonstrate how a variety of everyday applications can use sensed display adjustments to drive context-appropriate transitions, such as shifts between reading versus writing, displays of public versus personal information, face-to-face video versus screen sharing of documents in remote work, and other nuances of input and feedback contingent on display angle\u2014with continuous interactive responses tailored to each use case.<\/li><\/ul>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<article class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<div class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">PUBLICATION <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/tilt-responsive-techniques-for-digital-drawing-boards-uist-2020-paper\/\" data-bi-cN=\"Tilt-Responsive Techniques for Digital Drawing Boards \" data-external-link=\"false\" data-bi-aN=\"citation\" data-bi-type=\"annotated-link\" class=\"annotations__link font-weight-semibold text-decoration-none\"><span>Tilt-Responsive Techniques for Digital Drawing Boards <\/span>&nbsp;<span class=\"glyph-in-link glyph-append glyph-append-chevron-right\" aria-hidden=\"true\"><\/span><\/a>\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">Hugo Romat, Christopher Collins, Nathalie Henry Riche, Michel Pahud, Christian Holz, Adam Riddle, Bill Buxton, Ken Hinckley<\/p>\n\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Tilt-Responsive Techniques for Digital Drawing Boards\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/bovY2IdTbgE?feature=oembed&rel=0\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>During the long incubation and technical development of the distributed-systems advances discussed above, we kept circling back at odd intervals to another endeavor: we had outfitted the Microsoft Surface Studio with an extra sensor to detect its angle. The Microsoft Surface Studio is a 27\u201d screen that supports multi-touch and pen input and can be adjusted smoothly from a vertical display to a drafting table\u2013like 20 degrees. In this device, we saw a parallel between its use and people\u2019s behaviors and expectations outside the digital world.<\/p>\n\n\n\n<p>In everyday life, people naturally reposition objects, such as paper documents, to allow shared visibility, partial viewing, and even concealment. Such motions are completely natural and perhaps even subconscious. How we position an object depends on what we intend to do. For example, a doctor might hold a medical chart \u201cclose to the vest\u201d at first, but then turn it toward their patient when ready to share particular results. Similarly, the appropriate display orientation depends on the task and situation at hand. A vertical monitor makes for easier reading but not necessarily easier writing with a stylus. Angled drafting tables in a design studio encourage sketching and freeform brainstorming, but the preference when presenting refined versions of those same ideas may be a vertical screen. Display angle is not one size fits all. We wondered, could we tap into our most natural ways of mediating information exchange by sensing the tilt of a display?<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"541\" height=\"406\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4.png\" alt=\"Man working on Surface Studio\" class=\"wp-image-704128\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4.png 541w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4-300x225.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4-16x12.png 16w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4-80x60.png 80w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/UIST-Blog-fig-4-240x180.png 240w\" sizes=\"auto, (max-width: 541px) 100vw, 541px\" \/><figcaption>We explored system responses to the sensed tilt of an adjustable Microsoft Surface Studio display, such as during transitions from vertical to low-angled, drafting table\u2013like postures. This transforms the current application\u2019s user experience via continuous, interactive, sensor-driven transitions.<\/figcaption><\/figure>\n\n\n\n<p>By adding an off-the-shelf tilt sensor to the Microsoft Surface Studio, we discovered a series of designs, techniques, and interactions that can respond appropriately to the user\u2019s context of use, as sensed by the current display angle. In doing so, we begin to shift the burden of adapting the inputs, tools, modes, and graphical layout of applications from the user to the system. For example, one demonstration we built explores a teleconferencing scenario in which the typical talking-head video feed of \u201cperson-space\u201d appears when the screen is vertical but transitions to a shared document that users can mark up with a digital pen when the screen is tilted down like a drafting board. As the display tilts, we fade out the camera feed to let the user avoid unbecoming video angles. This also selectively focuses the remote audience\u2019s attention on the shared document rather than the video feed\u2014in effect, a remote way of steering a remote participant\u2019s attention in a manner analogous to angling a paper document toward a collaborator nearby.<\/p>\n\n\n\n<p>As the demo reel above illustrates, displays can respond to tilt by transitioning between reading and writing, public versus personal, authoring versus presenting, and other nuances of input and feedback\u2014and in ways that often can delight and entertain, as well.<\/p>\n\n\n\n<p>Amid our above distributed-systems research, this curiosity-driven work was a completely unrelated side project. <em>Or so we thought. <\/em>After we finished writing up tilt-responsive techniques for publication, we had an epiphany. As you adjust the angle of a display, you\u2019re:<\/p>\n\n\n\n<p>In the same physical location.<\/p>\n\n\n\n<p>On the same device.<\/p>\n\n\n\n<p>Using the same screen.<\/p>\n\n\n\n<p>Running the same application.<\/p>\n\n\n\n<p>But the new screen orientation doesn\u2019t afford the same tasks and activities\u2014you\u2019ve transitioned to a different <em>place.<\/em><\/p>\n\n\n\n<p>This subtly shifts your expectations of what is appropriate. And with just a tiny bit of awareness, well-designed software could provide a sort of intelligence by responding appropriately in kind. That is, the angle of a digital display is just another form of mobility. Here, the mobility is on the micro-level, moving from one <em>screen orientation<\/em> to another, as opposed to the more macro-level transitions that are the current focus of the Fleet system, such as moving from one device to another or across local and remote locations.<\/p>\n\n\n\n<h2 id=\"closing-thoughts\">Closing thoughts <\/h2>\n\n\n\n<p>We have discussed how the Fleet system explores new ways to think about mobility, and we\u2019ve shown how it builds on an exciting new distributed-systems technology known as Ambrosia. These technologies work to build and implement applications that not only go beyond the current device, but also unbind other dimensions of mobility\u2014the current user, the current application, the current time\u2014as well. Beyond that, as hinted at by our final example above, our research shows how sensors can bridge transitions from the current (sensed) context to another\u2014by responding appropriately to natural human activity.<\/p>\n\n\n\n<p>At the highest level, these advances hint at how devices can be better together\u2014complementing one another across an ecosystem of technologies\u2014instead of competing to add ever more complexity with each new device or service.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>For those of us who have transitioned to working from home over the course of the last year, we must navigate a strange new manifestation of mobility. Far-flung colleagues appear almost magically in grid format on a screen right in front of our faces, despite their remote locations. Yet at the same time, a document, [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":706870,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Ken Hinckley","user_id":"32521"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-704044","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[371909],"related-projects":[698833],"related-events":[697429],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-960x540.jpg\" class=\"img-object-cover\" alt=\"\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-1536x864.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-2048x1152.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/11\/1400x788_Uist_Still_no_logo-1920x1080.jpg 1920w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"Ken Hinckley","formattedDate":"November 19, 2020","formattedExcerpt":"For those of us who have transitioned to working from home over the course of the last year, we must navigate a strange new manifestation of mobility. Far-flung colleagues appear almost magically in grid format on a screen right in front of our faces, despite&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/704044","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=704044"}],"version-history":[{"count":14,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/704044\/revisions"}],"predecessor-version":[{"id":894483,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/704044\/revisions\/894483"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/706870"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=704044"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=704044"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=704044"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=704044"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=704044"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=704044"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=704044"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=704044"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=704044"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=704044"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=704044"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}