Socially Intelligent Meetings

Making (tele)presence work

The Socially Intelligent Meetings program explores social and technological interventions for making work-oriented telepresence effective, comfortable, adaptable, and robust. The guiding principles are that telepresence technologies should aim to drive social value rather than attempt to replicate face-to-face experiences, and that telepresence technologies should be designed with a holistic and flexible approach to workflows rather than fitted narrowly to tasks.

This program currently has four workstreams:

• All-Remote Meetings During COVID-19
• Mixed Reality and Robotic Telepresence
• Configuring Hybrid Meetings
• Social Devices

All-Remote Meetings During COVID-19

In common with companies around the world, from March in 2020 Microsoft employees entered a prolonged period of mandatory working from home due to the COVID-19 pandemic. We wondered what would happen to employees’ effectiveness and collegiality as all meetings went remote.

From April through August 2020 we recruited almost 1000 employees to take part in a large scale study involving two month diaries and weekly polls.

While there is a long history of research into all-remote meetings, there are still few definitive answers about fundamental issues of why video meetings are not the simple “next best thing to being there” that we have all assumed they would be or could be. This ‘natural experiment’ is highlighting the everyday reality of the difference in “being there” as against “being remote” in work situations, compelling us to think about how to manage the boundaries between presence and absence. This is an opportunity to re-examine the longstanding questions with some of the most wide-scale naturalistic data ever available.

We are currently analyzing the data. Outputs will be reported here as they become available.​

Current internal research

• See the Meetings during COVID-19 Project page.

Current external research

• Attention in remote moeetings
  • Collaborator: Brittany Duff
  • Department: Institute of Communications Research, University of Illinois at Urbana-Champaign

Mixed Reality and Robotic Telepresence: Pushing the envelope on presence and engagement

Live mediated collaboration is at an inflection point between traditional 2D video communication and Mixed reality and robotic telepresence. This research explores the boundaries of what can be achieved with high fidelity augmented or virtual reality avatars (which are hoped to provide the missing elements from the flat rectangular portals of traditional video communication) and robotic telepresence, which provides remote participants with independent mobility and physical social presence. This research also involves improving baseline understandings of geo-distributed collaboration such as attention, effectiveness, and inclusiveness.

Current internal research

Presence

• Multi-modal turn-taking behaviours with mixed reality avatars
• Truthfulness and influence of mixed reality avatars
• Meeting effectiveness of mixed reality avatars

Intern project

• Intern: Brennan Jones. Assistance from Yaying Zhang and Priscilla Wong
• VROOM: Virtual Robot Overlay for Online Meetings

Current external research

Microsoft PhD Scholarship: Interacting via Robotic Telepresence

• Student: Andriana Boudouraki
• Supervisors: Joel Fischer and Stuart Reeves
• Department: Mixed Reality Lab, Nottingham University

Microsoft PhD Scholarship – Cambridge University: Human avatars as co-workers

• Student: TBD
• Supervisor: Thomas Bohné
• Organisation: IfM Cyber-Human Lab, Cambridge University

Configuring Hybrid Meetings: Reasoning about people, the room, and collaboration patterns

The traditional video meeting stage is neither engaging nor inclusive because it is not geared to present customised and augmented social presence. Machine perception can reason about people, the room, and patterns of collaboration between people within and across connected rooms to produce meeting experiences that helps all users, local or remote, and inclusive of all abilities, understand and engage with the people and content of meetings. need to design compelling user experiences of augmented social presence, and we need to understand the communicative, organisational, and ethical fundamentals of how people will respond to and use such experiences.

Baseline research

Attention

• CHI 2020: Classification of Functional Attention in Video Meetings
• CHI 2020 LBW: Low Engagement As a Deliberate Practice of Remote Participants in Video Meetings
• CHI 2019 workshop paper: Meeting in the Middle: The Interpretation Gap Between People and Machines

Microsoft PhD Scholarship: Hybrid Meetings

• Student: Banu Saatci
• Supervisor: Clemens Nylandsted Klokmose
• Department:  School of Communication and Culture – Department of Digital Design and Information Studies, Aarhus University
• JCSCW2020: (Re)Configuring Hybrid Meetings: Moving from User-Centered Design to Meeting-Centered Design
• Collabtech2019: Hybrid Meetings in the Modern Workplace: Stories of Success and Failure

Current collaborative external research

Microsoft Productivity Grant: Supporting Hybrid Collaboration for the Teams of Tomorrow

Collaborators: Mirjam Augstein and Thomas Neumayr, University of Applied Sciences Upper Austria

Concept: Tomorrow’s information workers are increasingly employed in flexible work settings and oftentimes come upon situations where they engage in hybrid meetings and hybrid collaboration. Although such situations, with their dynamic interplay between co-located and remote collaborators, are increasingly supported by software and hardware tools, there are still significant research gaps related to the description and analysis of such settings (which would also allow for more targeted tool support). Thus, the full potential of existing tools such as the Microsoft Surface Hub with its software solutions for co-located (e.g., Shared Whiteboard) or remote (audio and video conferencing) collaboration in the collaborative settings of the future is not yet fully exploited and requires in-depth conceptual as well as technological research. The envisioned research endeavor includes 1) thorough grounding work on a descriptive framework for hybrid collaboration, a small part of which already exists and was published at ACM CSCW 2018 (Domino: A Descriptive Framework for Hybrid Collaboration and Coupling Styles in Partially Distributed Teams) and 2) technical work on a software prototype for the support of hybrid meetings and in-depth (on-the-fly as well as post-hoc) analysis functionalities based on Microsoft hardware and software tools and APIs. To draw conclusions, we will conduct an extensive qualitative user study.

• ECSCW2019 Workshop: Hybrid Collaboration – Moving Beyond Purely Co-Located or Remote Collaboration

Social Devices: Adaptable cross-device configurations to support ad hoc task needs

The Social Devices work stream explores how to enable devices to act as companions. We look to ways in which our computing devices can be dynamically aggregated and dis-aggregated to create user experiences that adapt to the changing social and activity demands we encounter throughout the day. Features using this research have shipped in Microsoft Teams as Mobile Sharing and Companion Experiences.

See our Garage Wall of Fame post:

• Mobile Sharing and Companion Experiences for Microsoft Teams Meetings.

See Microsoft Teams support documents at:

• Join > Join on multiple devices
• Mobile Sharing > Use your phone as a companion device in a meeting [Scroll down page]

These features grew out of several older explorations of video-calling and meetings.

Papers

• Beyond Presentations and Office Social: Slideware concept that enabled open access to shared interaction with slides across multiple devices.
• Ad hoc adaptability in video calling: A position paper in which we explore ad hoc adaptability across devices in video-calling. We note the current difficulty of even simple combinations, discuss design issues, briefly report on a study of ad hoc screen mirroring, and note future directions.
• SkypeBeam: An experimental system that enable lightweight multi-user wireless smartphone mirroring within a video call. The system enabled multiple smartphones to share both digital content as well as physical artefacts when mirroring the live view from the phone camera feed.

Current external research

HEIF Fellowship – Oxford: Multi-device configurations to support physical examinations in NHS video consultations

Collaborators: Lucas Seuren and Sara Shaw, Nuffield Department of Primary Care Health Sciences, University of Oxford

Concept: The UK National Healthcare Service (NHS) has the goal of replacing up to a third of all outpatient consultations with video consultations by 2024. With the patient and clinician no longer in the same room, physical examinations are challenging. Patients and carers in a video consultation have an active role in the examination while, at the same time must ‘show and tell’ their clinician what’s going on. Lucas Seuren and Sara Shaw at the University of Oxford are exploring how Microsoft Teams Mobile Sharing and Companion Experiences could enable a patient to talk to a clinician on a laptop or tablet and use their mobile phone as an additional camera, providing more opportunities to show their body and the environment, unlocking the potential of video consultations.