{"id":1116141,"date":"2025-01-08T18:45:16","date_gmt":"2025-01-09T02:45:16","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-academic-program&p=1116141"},"modified":"2025-06-08T12:03:13","modified_gmt":"2025-06-08T19:03:13","slug":"ntire-2025-vqe","status":"publish","type":"msr-academic-program","link":"https:\/\/www.microsoft.com\/en-us\/research\/academic-program\/ntire-2025-vqe\/","title":{"rendered":"CVPR 2025 Challenge on Video Quality Enhancement for Video Conferencing"},"content":{"rendered":"\n\n

NTIRE Workshop at CVPR 2025<\/p>\n\n\n\n\n\n\n

Introduction<\/h2>\n\n\n\n

Design a Video Quality Enhancement (VQE) model to enhance video quality in video conferencing scenarios by (a) improving lighting, (b) enhancing colors, (c) reducing noise, and (d) enhancing sharpness in video calls \u2013 giving a professional studio-like effect.<\/p>\n\n\n\n

We provide participants with a differentiable Video Quality Assessment (VQA) model, training and test videos. The participants submit enhanced videos which are used for evaluation in our crowdsourced framework.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Motivation<\/h2>\n\n\n\n

Light is a crucial component of visual expression and key to controlling texture, appearance, and composition. Professional photographers often have sophisticated studio-lights and reflectors to illuminate their subjects such that the true visual cues are expressed and captured. Similarly, tech-savvy users with modern desk setups employ a sophisticated combination of key and fill lights to give themselves control over their illumination and shadow characteristics.<\/p>\n\n\n\n

On the other hand, many users are constrained by their physical environment which may lead to poor positioning of ambient lighting or lack thereof. It is also commonplace to encounter flares, scattering, and specular reflections that may come from windows or mirror-like surfaces. Problems can be compounded by poor-quality cameras that may introduce sensor noise. This leads to poor visual experience during video calls and may have a negative impact on downstream tasks such as face detection and segmentation.<\/p>\n\n\n\n

The current production light correction solution in Microsoft Teams – called AutoAdjust, finds a global mapping of input to output colors which is updated sporadically. Since this mapping is global, the method is sometimes unable to find a correction that works well for both foreground and background. A better approach may be Image Relighting which only performs local correction in the foreground and gives users the option to dim their background – creating a pop-out effect. A possible side effect of local correction can be the reduction of local contrast which often serves as a proxy to convey depth in 2D images \u2013 thereby making people appear dull in some cases.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Registration<\/h2>\n\n\n\n

Participants are required to register on the CodaLab website (opens in new tab)<\/span><\/a>. Email used during registration will be used to add participants to our Slack workspace (opens in new tab)<\/span><\/a> which will be the default mode of day-to-day communication and where participants will submit their videos for subjective evaluation. For objective evaluation, please make your submission to the CodaLab website contains the correct (a) team name, (b) names, emails & affiliations of your team members, and (c) team captain.<\/p>\n\n\n\n

Please reach out to the challenge organizers at jain.varun@microsoft.com<\/a> if you need assistance with registration. <\/p>\n\n\n\n

<\/div>\n\n\n\n

Awards & Paper Submission<\/h2>\n\n\n\n

Top-ranking participants will receive a winner certificate. They will also be invited to submit their paper to NTIRE 2025 and participate in the challenge report – both of which will be included in the archived proceedings of the NTIRE workshop at CVPR 2025.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Citation<\/h2>\n\n\n\n

If you use our method, data, or code in your research, please cite:<\/p>\n\n\n\n

@inproceedings{ntire2025vqe,\n  title={{NTIRE} 2025 Challenge on Video Quality Enhancement for Video Conferencing: Datasets, Methods and Results},\n  author={Varun Jain and Zongwei Wu and Quan Zou and Louis Florentin and Henrik Turbell and Sandeep Siddhartha and Radu Timofte and others},\n  booktitle={Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},\n  year={2025}\n}<\/code><\/pre>\n\n\n\n
<\/div>\n\n\n\n
Subjective Results<\/h2>\n\n\n\n
We received 5 complete submissions for both the mid-point and final evaluations. For each team’s submission, we utilized our crowdsourced framework to evaluate their 3,000-video test set. This involved presenting human raters with 270K side-by-side video comparisons. Raters were asked to provide a preference rating on a scale of 1 to 5, where 1 and 5 represent strong preference for the left and right video respectively, and 2 and 4 represent weak preference. A rating of 3 indicates no preference. Furthermore, raters were prompted to specify if their decision was primarily influenced by (a) colors, (b) image brightness, or (c) skin tone.<\/p>\n\n\n\n
Here are the Bradley\u2013Terry scores for each team that maximize the likelihood of the observed P.910 voting:<\/p>\n\n\n\n
\"results_combined_mid_final\"<\/p>\n\n\n\n
Figure 5: Interval plots illustrating the mean P.910 Bradley-Terry scores and their corresponding 95% confidence intervals for the 5 submissions, input videos, and the provided baseline. (Top) Overall preference, and (bottom) factors influencing preference.<\/em><\/p>\n\n\n\n
<\/div>\n\n\n\n
<\/a><\/p>\n\n\n\n\n\n
Problem Statement<\/h2>\n\n\n\n
\"Linear<\/p>\n\n\n\n
Figure 1: P.910 study indicates that people prefer AutoAdjust over auto-corrected images (L), which are both preferred over the Image Relighting approach.<\/em><\/p>\n\n\n\n
We ran three P.910 [2]<\/a> studies totaling ~350,000 pairwise comparisons that measured people\u2019s preference for AutoAdjust (A) and Image Relighting (R*) over No effect (N) and images auto-corrected using Lightroom (L). We used the Bradley\u2013Terry model [1]<\/a> to compute per-method scores and observed that people preferred our current AutoAdjust more than any other method in all three studies.<\/p>\n\n\n\n
To take the next step towards achieving studio-grade video quality, one would need to (a) understand what people prefer and construct a differentiable Video Quality Assessment (VQA) metric, and (b) be able to train a Video Quality Enhancement (VQE) model that optimizes this metric. To solve the first problem, we used the abovementioned P.910 data and trained a VQA model that, given a pair of videos x1<\/sub> and x2<\/sub>, gives the probability of x1<\/sub> being better than x2<\/sub>. Given a test set, this information can be used to construct a ranking order of a given set of methods.<\/p>\n\n\n\n
We would now like to invite researchers to participate in a challenge aimed at developing Neural Processing Unit (NPU) friendly VQE models that leverage our trained VQA model to improve video quality.<\/p>\n\n\n\n
We look at the following properties of a video to judge its studio-grade quality:<\/p>\n\n\n\n
    \n
  1. Foreground illumination<\/strong> \u2013 the person (all body parts and clothing) should be optimally lit.<\/li>\n\n\n\n
  2. Natural colors<\/strong> \u2013 correction may make local or global color changes to make videos pleasing.<\/li>\n\n\n\n
  3. Temporal noise<\/strong> \u2013 correct for image and video encoding artefacts and sensor noise.<\/li>\n\n\n\n
  4. Sharpness <\/strong>\u2013 to make sure that correction algorithms do not introduce softness, the final image should at least be as sharp as the input.<\/li>\n<\/ol>\n\n\n\n
    <\/div>\n\n\n\n
    We realize that there may be many other aspects to a good video. For simplicity, we discount all except the ones mentioned above. Specifically, submissions are not judged on:<\/p>\n\n\n\n
      \n
    1. Egocentric motion \u2013 unstable camera may introduce sweeping motion or small vibrations that we do not aim to correct.<\/li>\n\n\n\n
    2. Masking of Background \u2013 spatial modification of background such as blurring or replacement with minimal changes to the foreground may improve subjective scores but we consider these out of domain. <\/li>\n\n\n\n
    3. Makeup and beautification \u2013 it is commonplace for users to apply beautification filters that alter their skin tone and facial features such as those found on Instagram and Snapchat. We do not aim for that aesthetic.<\/li>\n\n\n\n
    4. Removal of reflection on glasses and lens flare \u2013 despite being a common occurrence in video teleconference scenarios, we do not aim to remove reflections that may come from screens and other light sources onto users\u2019 glasses due to the risk associated with altering the users\u2019 eyes and gaze direction.<\/li>\n\n\n\n
    5. Avatars \u2013 A solution that synthesizes a photorealistic avatar of the subject and drives it based on the input video may score the highest in terms of noise, illumination and color. If it indeed minimizes the total cost function that takes into account all these factors, it is acceptable.<\/li>\n<\/ol>\n\n\n\n
      Solutions that significantly rely on altering properties other than what are discussed above will be asked to resubmit. Ensembles of models are allowed. Manual tweaking of hyperparameters for individual videos would lead to disqualification.<\/p>\n\n\n\n
      <\/div>\n\n\n\n
      Baseline Solution & Starter Code<\/h2>\n\n\n\n
      Since AutoAdjust was ranked higher than expert-edited images and Image Relighting methods, we will provide the participants with a baseline solution so that they can reproduce the AutoAdjust feature as currently shipped in Microsoft Teams.<\/p>\n\n\n\n
      More details can be found at our repository: https:\/\/github.com\/varunj\/cvpr-vqe (opens in new tab)<\/span><\/a><\/p>\n\n\n\n
      <\/div>\n\n\n\n
      Compute Constraints<\/h2>\n\n\n\n
      The goal is to have a computationally efficient solution that can be offloaded to NPU for CoreML inference. We establish a qualifying criterion of CoreML uint8 or fp16 models with at most 20.0×109<\/sup> MACs\/frame for an input resolution of 1280*720. We anticipate such a model to have a per frame processing time of ~9ms on an M1 Ultra powered Mac Studio and ~5ms on an M4 Pro powered Mac Mini for the given input resolution. Submissions not meeting this criterion will not be considered for evaluation.<\/p>\n\n\n\n
      <\/div>\n\n\n\n\n\n
      Dataset<\/h2>\n\n\n\n
      \"Image<\/p>\n\n\n\n
      Figure 2: Ground truth from (top) our synthetics framework, (bottom) our AutoAdjust solution. The top row shows the input with suboptimal foreground illumination which is fixed by adding a studio light setup in front of the subject which is simulated in synthetics and predicted via global changes in the real data<\/em>.<\/p>\n\n\n\n
      We provide 13,000 real videos for training, validation and testing of VQE methods. The videos are 10s long, encoded at 30 FPS and amount to a total of 3,900,000 frames. We keep 3,000 (23%) videos for testing and ranking submissions and make 10,000 (77%) available to the teams. They can choose to split it between training and validation sets as they desire. The teams are also free to use other publicly available open datasets but will have to be mindful about data drift.<\/p>\n\n\n\n
      In addition to this data, we also provide some paired data for supervised training as shown in Fig 2.<\/p>\n\n\n\n
      Note that it is possible, and encouraged, to learn a correction that is different from these ground truth labels and achieves a higher MOS score. Hence, these labels should be treated as suggestive improvements and not the global optima.<\/p>\n\n\n\n
        \n
      1. Real data\n