{"id":901248,"date":"2022-11-28T05:50:32","date_gmt":"2022-11-28T13:50:32","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-academic-program&p=901248"},"modified":"2023-10-14T10:56:10","modified_gmt":"2023-10-14T17:56:10","slug":"deep-noise-suppression-challenge-icassp-2023","status":"publish","type":"msr-academic-program","link":"https:\/\/www.microsoft.com\/en-us\/research\/academic-program\/deep-noise-suppression-challenge-icassp-2023\/","title":{"rendered":"ICASSP 2023 Deep Noise Suppression Challenge"},"content":{"rendered":"\n\n

Includes de-reverberation and suppression of interfering talkers for headset and speakerphone scenarios<\/p>\n\n\n\n\n\n\n

Program dates:<\/strong> November 2022\u2013August 2023<\/p>\n\n\n\n

See Results tab for final results evaluated on Blind Testset. Five papers were invited from 7 teams where three teams from Tencent agreed to submit one paper. These teams will submit 2-page paper in ICASSP 2023. For deadline, please see the timeline tab. These five teams will get an email with instructions on writing the 2-page paper. All teams are invited to submit their current and future work in DNS Challenge Special Issues of IEEE OJ-SP journal. More details on OJ-SP will be emailed to all participants in next few months. <\/em><\/p>\n\n\n\n

With decades of research, deep noise suppression (DNS) has advanced to improve subjective overall audio quality. However, we are still quite far from eliminating speech distortions which results from over-suppression of noise, reverberation and neighboring talkers etc. in real-world scenarios. Enhancing speech quality (i.e., eliminating speech distortion) and suppressing noise, reverberation and neighboring talkers are two trade-offs DNS models must handle. IEEE ICASSP 2023 Deep Noise Suppression (DNS) grand challenge is the 5th edition of Microsoft DNS challenges with focus on deep speech enhancement achieved by suppressing background noise, reverberation and neighboring talkers and enhancing the signal quality. This challenge invites researchers to develop real-time deep speech enhancement models for full band speech. Deep speech enhancement models submitted to challenge are supposed to do joint denoising and dereverberation in presence of neighboring (interfering) talkers. We will release the development test set for intermediate evaluation of challenge models and a blind test set for final evaluation for selecting top 5 models. Each test clips will have a corresponding enrollment clip (30s) for primary talker to enable the development of personalized models. Participants are encouraged to develop both personalized and non-personalized models to elucidate the benefits of personalized for deep speech enhancement models.<\/p>\n\n\n\n

The challenge has two tracks: (1) Headset (wired\/wireless headphone, earbuds such as airpods etc.) speech enhancement; (2) non-headset (speakerphone, built-in mic in laptop\/desktop\/mobile phone\/other meeting devices etc.) speech enhancement. Past challenges demonstrated that headset scenarios exhibit certain acoustic properties by dint of proximity of microphone to primary talker. Such acoustic properties can be leveraged to improve deep speech enhancement models with and without personalization. On the other hand, speakerphone cases exhibit different set of acoustic properties which motivates a separate track for non-headset scenarios. Participants can develop personalized as well as non-personalized (non-enrollment) deep speech enhancement models for both tracks. Blind test set will have paralinguistic test clips which will include standard forms of paralanguage including but not limited to The throat-clear, \u201dhmm\u201d or \u201dmhm\u201d, \u201dHuh?\u201d or \u201dwhat?, Gasps, Sighs, Moans and groans, Deceptive speech, Sincere speech, Speech with high-base, Speech with high-pitch, Speech with low-pitch, Confident speech, Tired speech (when talker is tired), Persuasive speech, Voice change mid-clip (i.e. mimicry in last 50% of the clip). Blind testset will also include emotional speech including but not limited to happy, sad, angry, yelling, crying, and laughter. Blind testset include real test clips with high reverberation, high reverberation with noise, and noise in presence of interfering talkers. Testset noises include but not limited to Office scenarios (Typing, AC, Door shutting, Eating\/munching, Copy machine, Squeaking chair, Notification sounds etc.), Home scenarios (Baby crying, Dogs, TV, Radiators, hair dryer, kitchen noise, running water etc.), Appliances (Washer Dryer, Dishwasher, Coffee maker, kitchen noise, Vacuum cleaner etc.), Fire alarm, Car, Inside parked car on busy road, In-car neighboring talkers, Traffic Road, Car noise (from machinery, control systems, turn signal etc.), Caf\u00b4e, Coffee machine, Blender, Background babble, Airport Announcements etc. <\/p>\n\n\n\n

In all test clips, we have only one primary talker in enrollment clip while noisy test clip may have noise, reverberation, one or more neighboring talkers in addition to a primary talker. The goal of deep speech enhancement model is to preserve primary speaker speech while suppressing everything else. We provide a flexible framework for synthesizing training datasets which allows participants to choose a subset of challenge dataset or add their own corpora to augmented challenge training dataset. The two tracks have different test sets using corresponding device types. Test clips are real-world recordings collected through crowd sourcing. Test set includes representative noisy scenarios relevant for video\/audio meetings in hybrid-work settings. The challenge overview paper will discuss the test set data collection in detail including list of devices, specifications sent to crowd-sourced workers, steps taken for quality assurance (QA) of test set. Test sets are selected to include speaker variety, device variety, different acoustic properties such as impulse response, direct-to-reverb-ratio (DRR), T60 which is achieved by changing the relative and absolute position of primary and interfering talkers, noise source and presence of reflecting surfaces etc. Along with training datasets and testsets, we also provide a baseline model (or enhanced clips) for both tracks. Track 1 and Track 2 both may have personalized or non-personalized models, but we only provide one baseline for each track.<\/p>\n\n\n\n

We also provide personalized P.835 subjective evaluation framework which is used for both tracks along with Word Accuracy (WAcc) Azure API. The personalized P.835 subjective evaluation framework is an improved version of framework used in the 4th DNS Challenge. Our subjective framework is a modified version of ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) P.835 which provides three scores for each test clip (and its corresponding enrollment clip) namely speech quality (SIG), background
noise quality (BAK), and overall quality (OVRL). This challenge aims to improve the subjective audio quality as measured by SIG, BAK and OVRL, and provide improvements in WAcc. Crowd-sourced workers doing subjective evaluation are instructed to rate interfering talkers as undesirable signal so the model which suppresses interfering talkers is rated higher. Similarly, WAcc ground-truth transcripts will only contain words spoken by the primary talker, thus treating interfering talker as undesirable signal. Enhanced test clips from past DNS Challenges had noticeable WAcc and SIG degradation due to over-suppression resulting in removal of primary talker\u2019s speech. We open-sourced DNSMOS P.835 which is a deep neural network (DNN) model for non-intrusive prediction of speech, background noise and overall quality of audio signal. DNSMOS aims to help participants with intermediate model evaluations. Participants need to register on challenge\u2019s CMT site which will be used for challenge communication: https:\/\/cmt3.research.microsoft.com\/DNSChallenge2023. Questions related to challenge can be sent to dns challenge@microsoft.com.<\/p>\n\n\n\n

The challenge paper is ICASSP 2022 Deep Noise Suppression Challenge (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Please NOTE<\/strong> that the intellectual property (IP) is not transferred to the challenge organizers, i.e., if code is shared\/submitted, the participants remain the owners of their code (when the code is made publicly available, an appropriate license should be added).<\/p>\n\n\n\n

Challenge Tracks<\/strong><\/p>\n\n\n\n

\u2022 This challenge has two tracks: Track-1: Headset DNS; Track-2: Speakerphone DNS.
\u2022 Each testclip in both tracks has enrollment clip with 30s duration. The enrollment speech can be noise-free or noisy with\/without reverberation. This facilitates multi-condition enrollment of primary talkers which serves as a measure of robustness for personalized models which use enrollment speech as additional input for denosing the testclips.
\u2022 Participants can choose to work on models with speaker enrollment or without it for one or both tracks. Each team can submit 1-4 models depending on what experiments they conduct. Each participating team can submit a maximum of one personalized and one non-personalized model for each track, e.g., a team can submit one personalized and one non-personalized model for Track 1 but not two personalized or two non-personalized model for Track 1. Similarly, a team can submit 4 models, personalized and non-personalized
model for Track 1 and personalized and non-personalized model for Track 2. All models for a track will be evaluated and ranked together i.e., both personalized and non-personalized models for Track 1 will go through one subjective evaluation. Similarly, for track 2, all models in one subjective evaluation.
\u2022 Participants are encouraged to conduct experiments with both personalized and non-personalized models to elucidate the benefits of personalization. Though, this is NOT a requirement for this challenge.
\u2022 Each track will have its own dev testset and blind testset, each consisting of 600 testclips. Thus, total of 2400 unique testclips is released in this challenge. The main difference between two tracks will be devices used for collecting the testsets, i.e., headsets for Track 1 or speakerphones for Track 2.<\/p>\n\n\n\n

Challenge Requirements<\/strong><\/p>\n\n\n\n

Failing to adhere to challenge rules will lead to disqualification from the challenge. <\/span><\/p>\n\n\n\n

Algorithmic <\/em><\/strong>laten<\/i><\/b><\/span>cy:<\/em><\/strong> The offset introduced by the whole processing chain including STFT, iSTFT, overlap-add, additional lookahead frames, etc., compared to just passing the signal through without modification. But this doesn’t include buffering latency.<\/p>\n\n\n\n