{"id":323162,"date":"2016-11-18T09:00:40","date_gmt":"2016-11-18T17:00:40","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=323162"},"modified":"2017-12-06T10:59:32","modified_gmt":"2017-12-06T18:59:32","slug":"keith-salmon-oregon-project","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/keith-salmon-oregon-project\/","title":{"rendered":"Interpreting visual art with sound for a more inclusive experience"},"content":{"rendered":"<p><em>By Vanessa Ho, Microsoft News Center Staff. Follow her on <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/x.com\/search?q=vanessa%20ho&src=typd\" target=\"_blank\">Twitter<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/em><\/p>\n<div id=\"attachment_323180\" style=\"width: 885px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-323180\" class=\"wp-image-323180 size-full\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/interpreting-visual-art-with-sound.jpg\" alt=\"Oregon Project\" width=\"875\" height=\"583\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/interpreting-visual-art-with-sound.jpg 875w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/interpreting-visual-art-with-sound-300x200.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/interpreting-visual-art-with-sound-768x512.jpg 768w\" sizes=\"auto, (max-width: 875px) 100vw, 875px\" \/><p id=\"caption-attachment-323180\" class=\"wp-caption-text\">Artist Keith Salmon, left, and Microsoft researcher Neel Joshi stand in front of \u201cThe Oregon Project\u201d at the King Street Station in Seattle. Photos by Dan DeLong<\/p><\/div>\n<p>A well-known Scottish landscape painter, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.keithsalmon.org\/\" target=\"_blank\">Keith Salmon<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> creates ethereal, moody abstracts of skylines and mountains inspired by the rugged highlands in his country. Trained in fine arts and sculpture, Salmon has broadened his techniques over the years as his eyesight has diminished, learning how to smash paint into texture and scratch pastels for scribbles to evoke less a depiction of place and more an exquisite experience of the wild.<\/p>\n<p>Last month, Salmon, who is legally blind, debuted an installation with innovative research from Microsoft that enriches his art even more, with proxemic audio to interpret two-dimensional images. Called \u201c<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/aka.ms\/oregonproject\" target=\"_blank\">The Oregon Project<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>,\u201d it uses four Kinects, 15 overhead speakers and 54 soundtracks to produce an acoustic and spatial interpretation of three drawings Salmon did of the beautifully remote Hells Canyon area in Oregon. The installation premiered in Seattle at the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.9e2seattle.org\/\" target=\"_blank\">9e2 exhibit<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of art and technology as a powerful new way for Salmon to create and for people with low vision to experience and enjoy visual art.<\/p>\n\t\t\t<div class=\"ms-grid \">\n\t\t\t<div class=\"ms-row\">\n\t\t\t\t\t<div  class=\"m-col-12-24\" >\n\t\t<p><iframe loading=\"lazy\" title=\"Interpreting visual art with sound for a more inclusive experience\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/HgSTLDgh0kc?feature=oembed&rel=0\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p><p>\t<\/div>\n\t\t<div  class=\"m-col-12-24\" >\n\t\t<\/p><p>\u201cIt would be great to think that in a few years, you might go to a national museum or art gallery and this technology would be there as part of the experience,\u201d says Salmon, who began to lose his sight in the \u201880s due to diabetic retinopathy. With his sight limited to a small amount of vision in one eye for many years, he\u2019s been able to adapt, paint and hike with his partner Anita Groves as his guide. You can watch the video with audio-description <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/youtu.be\/DvSFOLaiFOs\" target=\"_blank\">here<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p><p>\t<\/div>\n\t<\/p>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\n<p>In front of each drawing is a 3D space containing 18 soundtracks that can be played. The Kinects track movement to trigger different sounds, making the art lively and interactive for people with and without vision. From afar, observers can hear birdsong, rushing water and grass swishing in the wind \u2014 recorded in Hells Canyon by Salmon and his collaborators.<\/p>\n<p>Moving closer activates digital tones matched to Salmon\u2019s palette of blue, green, brown and ochre. Stand even closer and you can hear Salmon working in his studio as he scribbles pastels on paper, in an experience that parallels how a sighted person can see details when leaning in. The Kinects also track hand waves to change what you hear, helping observers become part of the piece and make their own acoustic mixes with movement. At the exhibit, adults triggered sounds primarily by walking, while kids jumped and waved their arms to produce a different blend of noise.<\/p>\n<p>\u201cAs a way of moving from an outer to inner layer to the drawings, natural sounds to industrial sounds works really well,\u201d Salmon says. \u201cIt has a wonderful, atmospheric, spacey, landscape-y feel that pulls the whole thing together.\u201d<\/p>\n<p>The sound system stems from an experimental project called Eyes-Free Art, created by Microsoft researchers <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/neel\/\" target=\"_blank\">Neel Joshi<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/merrie\/\" target=\"_blank\">Meredith Ringel Morris<\/a> and former Microsoft intern <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/homepage.cs.uiowa.edu\/~krector\/\" target=\"_blank\">Kyle Rector<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to explore how technology can help people with low vision access visual art.<\/p>\n<p>Many museums offer experiences for people with visual impairment, from detailed audio descriptions of art to special events where people can touch sculptures with gloved hands. But the researchers wanted to create a more personalized experience that was still communal, and something less directed and curated by docents.<\/p>\n<p>After exploring different ideas, they decided to create a Kinect-enabled sound system based on proxemics, the study of how people use space to define social interactions. And they wanted sound and visuals to be equally aesthetic and important, and rejected anything that felt like a \u201cbolted-on\u201d accessibility tool.<\/p>\n<p>\u201cVisual exploration is a personal experience and is based on distance \u2014 as I get closer, I see different types of details \u2014 so we wanted to mimic those ideas in audio,\u201d says Joshi, who specializes in computer vision and computational photography. He\u2019s also a painter and interactive-installation artist involved in the local arts community.<\/p>\n<div id=\"attachment_323174\" style=\"width: 385px\" class=\"wp-caption alignright\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-323174\" class=\"wp-image-323174\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/Kinect-enabled-visual.jpg\" alt=\"Kinect Neel Joshi\" width=\"375\" height=\"250\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/Kinect-enabled-visual.jpg 1401w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/Kinect-enabled-visual-300x200.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/Kinect-enabled-visual-768x512.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/Kinect-enabled-visual-1024x683.jpg 1024w\" sizes=\"auto, (max-width: 375px) 100vw, 375px\" \/><p id=\"caption-attachment-323174\" class=\"wp-caption-text\">Microsoft researcher Neel Joshi works on the Kinect-enabled sound system of \u201cThe Oregon Project.\u201d<\/p><\/div>\n<p>Last year, researchers invited people with low vision to test a prototype built with programmed music, narration and sounds acoustically interpreting paintings by Frida Kahlo, Henri Matisse and others. The testers overwhelmingly enjoyed the experience, and one woman, an art lover who had stopped going to galleries after losing her sight, cried in joy. But the team knew the system wasn\u2019t \u201cart.\u201d They knew they needed to do more.<\/p>\n<p>Connecting with local artists, Joshi began working with Salmon to create a new piece with Salmon\u2019s art and Microsoft research. They collaborated with Seattle filmmaker and photographer <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/alibipictures.com\/\" target=\"_blank\">Dan Thornton<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, Scottish sound engineer <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/www.linkedin.com\/in\/graham-byron-a349a582\" target=\"_blank\">Graham Byron<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and 9e2 organizer <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/boylanconversation.wordpress.com\/\" target=\"_blank\">John Boylan<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to display it at the nine-day 9e2 event of installations and performances on art, science and technology. Although this particular project has been experimental and there are no immediate plans to make this specific system available broadly to the public, researchers have filed a patent for Eyes-Free Art and hope to continue building more accessible, artistic experiences.<\/p>\n<p>Amos Miller, a Microsoft researcher with visual impairment who works with the company\u2019s Accessibility group, calls the installation \u201cinherently inclusive,\u201d allowing him and others to experience art personally and socially, and in the way the artist intended.<\/p>\n<p>\u201cIt\u2019s evidence that you can create really positive, inclusive experiences,\u201d says Miller, who is part of the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/news.microsoft.com\/en-gb\/2015\/11\/25\/cities-unlocked-a-voyage-of-discovery\/#ye1MlwhtEuUHvwlk.97\" target=\"_blank\">Cities Unlocked<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> project, which creates 3D soundscapes to help people with low vision navigate their surroundings.<\/p>\n<p>For Sheri Richardson, president of Seattle advocacy group <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.visionlossconnections.org\/\" target=\"_blank\">Vision Loss Connections<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and blind since infancy, the installation was a vibrant, sensory way to explore art from different perspectives and not rely on someone else\u2019s interpretation. She especially liked being able to visualize the artist in the mountains with sounds from Hells Canyon.<\/p>\n<p>\u201cThe technology of the Kinects and the sound moving with you as you move into the paintings really makes it experiential and brings out the emotions,\u201d says Richardson.<\/p>\n<div id=\"attachment_323171\" style=\"width: 385px\" class=\"wp-caption alignleft\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-323171\" class=\"wp-image-323171\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/third-photo.jpg\" alt=\"Oregon Project Keith Salmon\" width=\"375\" height=\"250\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/third-photo.jpg 909w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/third-photo-300x200.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/11\/third-photo-768x512.jpg 768w\" sizes=\"auto, (max-width: 375px) 100vw, 375px\" \/><p id=\"caption-attachment-323171\" class=\"wp-caption-text\">Artist Keith Salmon in front of his drawings of Hells Canyon in &#8220;The Oregon Project&#8221;<\/p><\/div>\n<p>For Salmon, whose limited vision began to further deteriorate a few years ago, Eyes-Free Art has given him hope to continue his artistic career for many more years.<\/p>\n<p>\u201cIt\u2019s been amazing,\u201d he says. \u201cI\u2019ve now got a future. If my sight carries on getting worse, I can continue making art and draw with sound. It\u2019s a case of learning and adapting.\u201d<\/p>\n<p>\u201cThe Oregon Project\u201d will be on display at the Tent Gallery at the Edinburgh College of Art in Scotland in April of 2017.<\/p>\n<p>Learn more about Salmon and \u201cThe Oregon Project\u201d at Microsoft\u2019s <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.facebook.com\/microsoft\" target=\"_blank\">Facebook<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/www.instagram.com\/microsoft\" target=\"_blank\">Instagram<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"https:\/\/x.com\/Microsoft\" target=\"_blank\">Twitter<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> pages.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Vanessa Ho, Microsoft News Center Staff. Follow her on Twitter. A well-known Scottish landscape painter, Keith Salmon creates ethereal, moody abstracts of skylines and mountains inspired by the rugged highlands in his country. Trained in fine arts and sculpture, Salmon has broadened his techniques over the years as his eyesight has diminished, learning how [&hellip;]<\/p>\n","protected":false},"author":32627,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194468,194471],"tags":[219947,219932,219938,219941,219929,219944,219935],"research-area":[13562,13551],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-323162","post","type-post","status-publish","format-standard","hentry","category-audio","category-computer-vision","tag-3d-soundscapes","tag-9e2-exhibit","tag-cities-unlocked-project","tag-eyes-free-art","tag-keith-salmon","tag-the-oregon-project","tag-vision-loss-connections","msr-research-area-computer-vision","msr-research-area-graphics-and-multimedia","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144928,615681],"related-projects":[443373],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"November 18, 2016","formattedExcerpt":"By Vanessa Ho, Microsoft News Center Staff. Follow her on Twitter. A well-known Scottish landscape painter, Keith Salmon creates ethereal, moody abstracts of skylines and mountains inspired by the rugged highlands in his country. Trained in fine arts and sculpture, Salmon has broadened his techniques&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/323162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/32627"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=323162"}],"version-history":[{"count":34,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/323162\/revisions"}],"predecessor-version":[{"id":324065,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/323162\/revisions\/324065"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=323162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=323162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=323162"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=323162"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=323162"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=323162"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=323162"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=323162"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=323162"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=323162"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=323162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}