Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and depth cameras, our implemented prototypes explore sensing and interaction techniques that offer a range of compelling capabilities.<\/p>\n
User scenarios include context-aware privacy, gestural audio-visual control, and co-opting natural body language as context to drive animated avatars for “camera-off” scenarios in remote work\u2013or even to co-opt (oft-subconscious) head movements such as dodging attacks in video games to enhance the gameplay experience.<\/p>\n
Drawing from the literature and other frameworks, we situate our prototypes and related techniques in a design space across the dual dimensions of (1) type of input (touch, mid-air, or head orientation\/motion); and (2) the context of user action (application, body, or environment). In particular, interactions that combine multiple inputs and contexts at the same time offer a rich design space of headphone-situated wearable interactions and sensing techniques.<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and […]<\/p>\n","protected":false},"featured_media":940875,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_publishername":"ACM","msr_publisher_other":"","msr_booktitle":"","msr_chapter":"","msr_edition":"","msr_editors":"","msr_how_published":"","msr_isbn":"","msr_issue":"","msr_journal":"","msr_number":"","msr_organization":"","msr_pages_string":"","msr_page_range_start":"","msr_page_range_end":"","msr_series":"","msr_volume":"","msr_copyright":"Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and\/or a fee. Request permissions from Permissions@acm.org. DIS '23, July 10\u201314, 2023, Pittsburgh, PA, USA \u00a9 2023 Copyright is held by the owner\/author(s). Publication rights licensed to ACM. ACM 978-1-4503-9893-0\/23\/07\u2026$15.00 https:\/\/doi.org\/10.1145\/3563657.3596022 Payod Panda, Molly Jane Nicholas, David Nguyen, Eyal Ofek, Michel Pahud, Sean Rintel, Mar Gonzalez-Franco, Ken Hinckley, and Jaron Lanier. 2023. Beyond Audio: Towards a Design Space of Headphones as a Site for Interaction and Sensing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS '23). Association for Computing Machinery, New York, NY, USA, 904\u2013916. https:\/\/doi.org\/10.1145\/3563657.3596022","msr_conference_name":"DIS'23: ACM Designing Interactive Systems Conference 2023","msr_doi":"","msr_arxiv_id":"","msr_s2_paper_id":"","msr_mag_id":"","msr_pubmed_id":"","msr_other_authors":"","msr_other_contributors":"","msr_speaker":"","msr_award":"","msr_affiliation":"","msr_institution":"","msr_host":"","msr_version":"","msr_duration":"","msr_original_fields_of_study":"","msr_release_tracker_id":"","msr_s2_match_type":"","msr_citation_count_updated":"","msr_published_date":"2023-7-1","msr_highlight_text":"Best Paper Award","msr_notes":"","msr_longbiography":"","msr_publicationurl":"","msr_external_url":"","msr_secondary_video_url":"","msr_conference_url":"https:\/\/dis.acm.org\/2023\/","msr_journal_url":"","msr_s2_pdf_url":"","msr_year":0,"msr_citation_count":0,"msr_influential_citations":0,"msr_reference_count":0,"msr_s2_match_confidence":0,"msr_microsoftintellectualproperty":true,"msr_s2_open_access":false,"msr_s2_author_ids":[],"msr_pub_ids":[],"msr_hide_image_in_river":null,"footnotes":""},"msr-research-highlight":[246574],"research-area":[13554],"msr-publication-type":[193716],"msr-publisher":[],"msr-focus-area":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[248485],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-938721","msr-research-item","type-msr-research-item","status-publish","has-post-thumbnail","hentry","msr-research-highlight-award","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-field-of-study-human-computer-interaction"],"msr_publishername":"ACM","msr_edition":"","msr_affiliation":"","msr_published_date":"2023-7-1","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"Best Paper Award","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"file","viewUrl":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2023\/05\/dis23-64-2.pdf","id":"950457","title":"dis23-64-2","label_id":"243132","label":0},{"type":"url","viewUrl":"false","id":"false","title":"https:\/\/doi.org\/10.1145\/3563657.3596022","label_id":"243106","label":0}],"msr_related_uploader":"","msr_citation_count":0,"msr_citation_count_updated":"","msr_s2_paper_id":"","msr_influential_citations":0,"msr_reference_count":0,"msr_arxiv_id":"","msr_s2_author_ids":[],"msr_s2_open_access":false,"msr_s2_pdf_url":null,"msr_attachments":[{"id":950457,"url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2023\/06\/dis23-64-2.pdf"},{"id":940878,"url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2023\/05\/Beyond_Audio__Headphones_as_Input-1.pdf"}],"msr-author-ordering":[{"type":"user_nicename","value":"Payod Panda","user_id":44104,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Payod Panda"},{"type":"text","value":"Molly Jane Nicholas","user_id":0,"rest_url":false},{"type":"text","value":"David Nguyen","user_id":0,"rest_url":false},{"type":"text","value":"Eyal Ofek","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Michel Pahud","user_id":33007,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Michel Pahud"},{"type":"user_nicename","value":"Sean Rintel","user_id":33579,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Sean Rintel"},{"type":"text","value":"Mar Gonzalez Franco","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Ken Hinckley","user_id":32521,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Ken Hinckley"},{"type":"user_nicename","value":"Jaron Lanier","user_id":32148,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jaron Lanier"}],"msr_impact_theme":[],"msr_research_lab":[199561,199565],"msr_event":[],"msr_group":[379814,1105932],"msr_project":[984894,937905,698833,483294],"publication":[],"video":[],"msr-tool":[],"msr_publication_type":"inproceedings","related_content":{"projects":[{"ID":984894,"post_title":"Understanding Virtual & Hybrid Meetings","post_name":"understanding-virtual-hybrid-meetings","post_type":"msr-project","post_date":"2023-11-15 10:15:50","post_modified":"2026-03-24 16:35:39","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/understanding-virtual-hybrid-meetings\/","post_excerpt":"Formative studies in distributed meeting behavior and technologies This is a collection of studies in distributed meeting behavior and technologies, covering virtual meetings, hybrid meetings, mobile robotic telepresence, and avatars. See Publications.","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/984894"}]}},{"ID":937905,"post_title":"Transcendence","post_name":"transcendence","post_type":"msr-project","post_date":"2023-05-01 15:33:11","post_modified":"2024-04-05 08:00:06","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/transcendence\/","post_excerpt":"Reinventing how we work together The Transcendence Project at Microsoft Research is reimagining interaction, productivity, and collaboration, harnessing the power of AI to transcend space, time, and modality, and redefine how we work together in the future.","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/937905"}]}},{"ID":698833,"post_title":"SurfaceFleet","post_name":"surfacefleet","post_type":"msr-project","post_date":"2020-10-16 15:09:07","post_modified":"2023-04-21 22:31:42","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/surfacefleet\/","post_excerpt":"SurfaceFleet is a system and toolkit that uses resilient and performant distributed programming techniques to explore cross-device user experiences. With appropriate design, these technologies afford mobility of user activity unbounded by device, application, user, and time. The vision of the project is to enable a future where an ecosystem of technologies seamlessly transition user activity from one place to another, whether that \u201cplace\u201d takes the form of a literal location, a different device form-factor, the…","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/698833"}]}},{"ID":483294,"post_title":"Intentional Meetings","post_name":"intentional-meetings","post_type":"msr-project","post_date":"2020-02-13 04:03:50","post_modified":"2026-03-24 16:30:04","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/intentional-meetings\/","post_excerpt":"Exploring how to make remote and hybrid telepresence meetings engaging, effective, adaptive, and inclusive.","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/483294"}]}}]},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/938721","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":6,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/938721\/revisions"}],"predecessor-version":[{"id":1162729,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/938721\/revisions\/1162729"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/940875"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=938721"}],"wp:term":[{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=938721"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=938721"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=938721"},{"taxonomy":"msr-publisher","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publisher?post=938721"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=938721"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=938721"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=938721"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=938721"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=938721"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=938721"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=938721"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=938721"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}