{"id":1114599,"date":"2025-01-03T23:11:11","date_gmt":"2025-01-04T07:11:11","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-academic-program&p=1114599"},"modified":"2025-06-11T06:42:22","modified_gmt":"2025-06-11T13:42:22","slug":"photorealistic-avatar-challenge-cvpr-2025","status":"publish","type":"msr-academic-program","link":"https:\/\/www.microsoft.com\/en-us\/research\/academic-program\/photorealistic-avatar-challenge-cvpr-2025\/","title":{"rendered":"Photorealistic Avatar Challenge CVPR 2025"},"content":{"rendered":"\n\n

<\/p>\n\n\n\n\n\n\n

Program dates<\/strong>: January 15, 2025, to June 10, 2025<\/p>\n\n\n\n

The CVPR 2025 Photorealistic Avatar Challenge is intended to stimulate research in the field of photorealistic avatars. The challenge provides a test set and methodology to subjectively evaluate photorealistic avatars for news anchor and telecommunication scenarios. The evaluation task is audio\/video driven self-reenactment. Test subjects will be sitting or standing but only the upper half of the body is rendered for tracks 1 and 2; track 3 will have head-only video. Speech, facial emotions, head turning, and hand gestures sequences are included. The inputs to the challenge are enrollment and test video clips. The output are self-reenactment video clips which we will evaluate with our subjective test framework and are used as the challenge metric.<\/p>\n\n\n\n

Challenge tracks<\/h3>\n\n\n\n

There are three challenge tracks:<\/p>\n\n\n\n

    \n
  1. Half-body real-time<\/li>\n\n\n\n
  2. Half-body non-real-time<\/li>\n\n\n\n
  3. Head-only non-real-time – NEW!<\/strong><\/li>\n<\/ol>\n\n\n\n

    Registration procedure<\/h3>\n\n\n\n

    Registration is open!<\/strong> To register for the challenge,\u202fparticipants are required to email the CVPR Photorealistic Avatar Challenge organizers avatar_challenge@microsoft.com<\/a> with the name of their team members, emails, affiliations, team name, track(s) participating in, team captain, and tentative paper title. Participants also need to register on the Challenge CMT (opens in new tab)<\/span><\/a> site where they can submit the enhanced clips. Registration data is captured and stored in the US.<\/p>\n\n\n\n

    Submission instructions<\/h3>\n\n\n\n

    Please use Microsoft Conference Management Toolkit (opens in new tab)<\/span><\/a> for submitting the results. After logging in, complete the following steps to submit the results<\/p>\n\n\n\n

      \n
    1. Choose \u201cCreate new submission\u201d in the Author Console.<\/li>\n\n\n\n
    2. Enter the title, abstract, and co-authors, and upload a lastname<\/em>.txt file (can be empty or contain additional information regarding the submission).<\/li>\n\n\n\n
    3. Compress the enhanced results files to a single lastname<\/em>.zip file, retaining the same folder and file names as the blind test set (max file size: 1.8 GB).<\/li>\n\n\n\n
    4. After creating the submission, return to the \u201cAuthor Console\u201d (by clicking on \u201cSubmissions\u201d at the top of the page) and upload the lastname<\/em>.zip file via \u201cUpload Supplementary Material\u201d.<\/li>\n<\/ol>\n\n\n\n

      Contact us:<\/strong> For questions, please contact <\/a>avatar_challenge@microsoft.com<\/a> <\/p>\n\n\n\n

      Sample images from some of the clips in the test set (tracks 1 and 2)<\/h3>\n\n\n\n
      \"Video<\/td>\"id3\"<\/td>\"id1\"<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      Sample image from some of the clips in the test set (track 3)<\/h2>\n\n\n\n
      \"Track<\/figure>\n\n\n\n\n\n

      Photorealistic avatars are human avatars that look, move, and talk like real people. Photorealistic avatars can be used for various applications, such as telecommunication, health care, education, retail and e-commerce, and entertainment. There has been a significant increase in research and publications about photorealistic avatars. CVPR 2024 included 36 papers on photorealistic avatars using a variety of methods. While there are a few common test sets that are used, there is no one test set that all use. Also, only quantitative metrics are used (PSNR, SSIM, and LPIPS) which have well known limitations like weak correlation to subjective realism. None of the CVPR 2024 photorealistic avatar papers included subjective tests for avatar performance, in part because this is a challenging task but also because there is no standardized or readily available method to do so.<\/p>\n\n\n\n

      In [1] we define and implement the first multidimensional measurement of photorealistic avatar quality of experience. We provide an open source implementation of the subjective test framework based on our extension to ITU-T P.910. We include subjective measurements of avatar realism, affinity, trust, comfortableness using, comfortableness interacting, appropriateness for work, creepiness, formality, resemblance to the person, emotion accuracy, and gesture accuracy. We show that except for resemblance the correlation of these subjective metrics to PSNR, SSIM, and LPIPS is weak; the correlation for emotion accuracy is moderate. For example, the avatar with the best PSNR, SSIM, and LPIPS in [1] (MS3_0 in Table 4) is average in terms of the subjective metrics. In other words, the objective metrics PSNR, SSIM, and LPIPS cannot be used to accurately stack rank the subjective performance of photorealistic avatars. The crowdsourced subjective test framework we have developed has been shown to be highly reproducible and accurate compared to a panel of experts. We also found that for avatars above a certain level of realism (mean opinion score > 2.5 in a 1-5 scale) these measured dimensions are highly correlated. In particular, for photorealistic avatars there is a linear relationship between avatar affinity and realism.<\/p>\n\n\n\n

      This challenge will provide a test set and methodology to subjectively evaluate photorealistic avatars for news anchor and telecommunication scenarios. Test subjects will be sitting or standing but only the upper half of the body is rendered. Speech, facial emotions, head turning, and hand gestures sequences are included.<\/p>\n\n\n\n

      Tracks 1 and 2<\/h3>\n\n\n\n

      The challenge includes two tracks for half-body, (1) real-time and (2) non-real-time. Tracks 1 and 2 are evaluated on the same test set and target a face and upper torso avatar including hands. The real-time track must be evaluated on a NVIDIA RTX 4090 or equivalent GPU and must render the avatar at 1080p 30 FPS (each frame must be captured, processed, and rendered in less than 33 ms total). Teams can submit entries to tracks 1 and 2, but for the non-real-time track the total processing time per frame must be greater than 33 ms. The avatars must be causal and not use any future frames in the rendering of the current frame. <\/p>\n\n\n\n

      The challenge metric<\/strong> is the mean of the subjective dimensions defined in [1] with the addition of gesture accuracy. Specifically:<\/p>\n\n\n\n

      Challenge metric<\/strong> = mean(realism, resemblance to the person, emotion accuracy, gesture accuracy)<\/p>\n\n\n\n

      At least N=30 ratings per clip will be used for ranking the entries, and statistical tests will be done to determine ties. Additional metrics defined in [1] will be measured but will not be used as the challenge metric.<\/p>\n\n\n\n

      The test set will consist of a data from 10 people. It will consist of 5 males, 5 females, with a mix of Caucasian, Asian, and Black races for diversity. Each person has a 60 second enrollment clip with head motions, expressions, and speaking. The enrollment clip may be captured on different days as the test set. The test set consists of a 10 second speaking clip, a 15 second non-speaking clip that includes 6 emotions (happy, sad, anger, fear, surprised, and disgust), and a 20 second clip that includes hand gestures and head turns up to 90 degrees. An initial test set of 5 subjects will be provided at the beginning of the challenge, and the final test set will be provided 1 week before challenge submissions. The enrollment and test clips will be captured with a white background. Challenge submissions must be provided in 1080p mp4 format with CRF=17. The clips will be captured on recent iPhone cameras and will be provided in 4K 30 FPS. The baseline avatars will be [2] (which does well with gestures but is not as realistic as live video) and [3] (which is very realistic but does not handle gestures).<\/p>\n\n\n\n

      The test set is available at CVPR 2025 Photorealistic Avatar Challenge test set (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

      Challenge input\/output<\/h3>\n\n\n\n

      The general process for the challenge is as follows:<\/p>\n\n\n\n

        \n
      1. Create an avatar model<\/li>\n\n\n\n
      2. For each enrollment clip\n