Audio Cameras for Audio-Visual Scene Analysis

June 3, 2009
Ramani Duraiswami and Adam O'Donovan | The University of Maryland, College Park, MD

Computer vision has been able to use images to reason about scenes. We develop a new device, the audio camera, that displays images of the auditory energy arriving at a particular point, much in the same way that a visual camera displays this information for light. This device is based on the spherical microphone array, and describe how we were able to achieve frame-rate operation using graphical processing. Next, we consider the combination of the audio-camera with visual cameras. Based on the observation that audio cameras also produce central projection images, we are able to jointly calibrate audio and visual cameras. This allows applications such as

Using the epipolar constraint to guide beamforming in noisy environments
Do audio-visual image transfers
Image room and concert-hall acoustics
Image-based matched filter beamforming and dereverberation

The audio camera design, has also been iterated to make the design robust, and easy to deploy.

Time permitting, we will also briefly touch on other research topics in our research group, including audio scene rendering; the fast multipole method, and fast kernel algorithms; scientific computing on GPUs.

Speaker Details

Ramani Duraiswami is an Associate Professor in the Department of Computer Science and the Institute for Advanced Computer Studies, at the University of Maryland, College Park. He obtained the B.Tech degree from IIT Bombay and a Ph.D. from Johns He directs research at the Perceptual Interfaces and Reality Laboratory there. His current research interests can broadly be categorized as lying in the three areas of Audio for Virtual Reality and Human Computer Interaction, Scientific Computing (with a focus on the Fast Multipole Method and GPU computing), and Computational Machine Learning and Vision. He is a Member of the IEEE Audio and Electroacoustics Technical Committee, Vice Chair of the IEEE Signal Processing Washington Chapter, and was till recently Associate Editor of the ACM Transactions on Applied Perception. More information on Dr. Duraiswami’s research and publications can be obtained from .

Adam O’Donovan is a Ph.D. candidate and graduate research assistant at the Department of Computer Science at the University of Maryland. He has B.S. degrees in Physics and Computer Science from the University of Maryland. Mr. O’Donovan received the NVIDIA fellowship for 2008-2009, and the University of Maryland Prime fellowship in 2007-2008. He has interned at NASA, Microsoft Research and Avaya. His research interests are in computational audition and interactive systems. Adam’s website is at http://adamod.googlepages.com