Socially Intelligent Meetings

How can we make geo-distributed collaboration socially engaging, adaptive, and inclusive?

Video meetings challenge productivity value because they are communicatively asymmetrical – spatial production and reception of talk are separated and constrained. Hybrid meetings (in which at least one endpoint is a group of people in a room) exacerbate asymmetry of video meetings because local room group dynamics are more immediately conversationally actionable than endpoint-to-endpoint dynamics. So although we can see and hear one another on almost any device, content and control are isolated on devices or physically limited to the room. Social nuances and engagement are hard for remote participants. We know that people can adapt technology to their communication needs, and they can also adapt their communication to technological enablements and constraints (leading to ‘technologized interaction‘) but the potential of video-mediated communication is greater than simply replicating what we can do in a face-to-face meeting.  The Socially Intelligent Meetings workstream explores how to use technology to augment the social intelligence of participants in video meetings. By so doing, we may be able to reduce the need for energy-intensive travel and improve the productivity of work anytime, anywhere, and on any-device.

This program of work currently has three strands

• Mixed Reality and Robotic Telepresence
• Configuring Hybrid Meetings
• Social Devices

Mixed Reality and Robotic Telepresence: Pushing the envelope on presence and engagement

Live mediated collaboration is at an inflection point between traditional 2D video communication and Mixed reality and robotic telepresence. This research explores the boundaries of what can be achieved with high fidelity augmented or virtual reality avatars (which are hoped to provide the missing elements from the flat rectangular portals of traditional video communication) and robotic telepresence, which provides remote participants with independent mobility and physical social presence. This research also involves improving baseline understandings of geo-distributed collaboration such as attention, effectiveness, and inclusiveness.

Current internal research

Presence

• Multi-modal turn-taking behaviours with mixed reality avatars
• Truthfulness and influence of mixed reality avatars
• Meeting effectiveness of mixed reality avatars

Intern project

• Intern: Brennan Jones. Assistance from Yaying Zhang and Priscilla Wong
• VROOM: Virtual Robot Overlay for Online Meetings

Current external research

Microsoft PhD Scholarship: Interacting via Robotic Telepresence

• Student: Andriana Boudouraki
• Supervisors: Joel Fischer and Stuart Reeves
• Department: Mixed Reality Lab, Nottingham University

Microsoft PhD Scholarship – Cambridge University: Human avatars as co-workers

• Student: TBD
• Supervisor: Thomas Bohné
• Organisation: IfM Cyber-Human Lab, Cambridge University

Configuring Hybrid Meetings: Reasoning about people, the room, and collaboration patterns

The traditional video meeting stage is neither engaging nor inclusive because it is not geared to present customised and augmented social presence. Machine perception can reason about people, the room, and patterns of collaboration between people within and across connected rooms to produce meeting experiences that helps all users, local or remote, and inclusive of all abilities, understand and engage with the people and content of meetings. need to design compelling user experiences of augmented social presence, and we need to understand the communicative, organisational, and ethical fundamentals of how people will respond to and use such experiences.

Baseline research

Attention

• CHI 2020: Classification of Functional Attention in Video Meetings
• CHI 2020 LBW: Low Engagement As a Deliberate Practice of Remote Participants in Video Meetings
• CHI 2019 workshop paper: Meeting in the Middle: The Interpretation Gap Between People and Machines

Current collaborative external research

Microsoft Productivity Grant: Supporting Hybrid Collaboration for the Teams of Tomorrow

Collaborators: Mirjam Augstein and Thomas Neumayr, University of Applied Sciences Upper Austria

Concept: Tomorrow’s information workers are increasingly employed in flexible work settings and oftentimes come upon situations where they engage in hybrid meetings and hybrid collaboration. Although such situations, with their dynamic interplay between co-located and remote collaborators, are increasingly supported by software and hardware tools, there are still significant research gaps related to the description and analysis of such settings (which would also allow for more targeted tool support). Thus, the full potential of existing tools such as the Microsoft Surface Hub with its software solutions for co-located (e.g., Shared Whiteboard) or remote (audio and video conferencing) collaboration in the collaborative settings of the future is not yet fully exploited and requires in-depth conceptual as well as technological research. The envisioned research endeavor includes 1) thorough grounding work on a descriptive framework for hybrid collaboration, a small part of which already exists and was published at ACM CSCW 2018 (Domino: A Descriptive Framework for Hybrid Collaboration and Coupling Styles in Partially Distributed Teams) and 2) technical work on a software prototype for the support of hybrid meetings and in-depth (on-the-fly as well as post-hoc) analysis functionalities based on Microsoft hardware and software tools and APIs. To draw conclusions, we will conduct an extensive qualitative user study.

• ECSCW2019 Workshop: Hybrid Collaboration – Moving Beyond Purely Co-Located or Remote Collaboration

Microsoft PhD Scholarship: Hybrid Meetings

• Student: Banu Saatci
• Supervisor: Clemens Nylandsted Klokmose
• Department:  School of Communication and Culture – Department of Digital Design and Information Studies, Aarhus University
• Collabtech2019: Hybrid Meetings in the Modern Workplace: Stories of Success and Failure

Social Devices: Adaptable cross-device configurations to support ad hoc task needs

The Social Devices work stream explores how to enable devices to act as companions. We look to ways in which our computing devices can be dynamically aggregated and dis-aggregated to create user experiences that adapt to the changing social and activity demands we encounter throughout the day. Features using this research have shipped in Microsoft Teams as Mobile Sharing and Companion Experiences.

See our Garage Wall of Fame post:

• Mobile Sharing and Companion Experiences for Microsoft Teams Meetings.

See Microsoft Teams support documents at:

• Join > Join on multiple devices
• Mobile Sharing > Use your phone as a companion device in a meeting [Scroll down page]

These features grew out of several older explorations of video-calling and meetings.

Papers

• Beyond Presentations and Office Social: Slideware concept that enabled open access to shared interaction with slides across multiple devices.
• Ad hoc adaptability in video calling: A position paper in which we explore ad hoc adaptability across devices in video-calling. We note the current difficulty of even simple combinations, discuss design issues, briefly report on a study of ad hoc screen mirroring, and note future directions.
• SkypeBeam: An experimental system that enable lightweight multi-user wireless smartphone mirroring within a video call. The system enabled multiple smartphones to share both digital content as well as physical artefacts when mirroring the live view from the phone camera feed.

Current external research

HEIF Fellowship – Oxford: Multi-device configurations to support physical examinations in NHS video consultations

Collaborators: Lucas Seuren and Sara Shaw, Nuffield Department of Primary Care Health Sciences, University of Oxford

Concept: The UK National Healthcare Service (NHS) has the goal of replacing up to a third of all outpatient consultations with video consultations by 2024. With the patient and clinician no longer in the same room, physical examinations are challenging. Patients and carers in a video consultation have an active role in the examination while, at the same time must ‘show and tell’ their clinician what’s going on. Lucas Seuren and Sara Shaw at the University of Oxford are exploring how Microsoft Teams Mobile Sharing and Companion Experiences could enable a patient to talk to a clinician on a laptop or tablet and use their mobile phone as an additional camera, providing more opportunities to show their body and the environment, unlocking the potential of video consultations.