Offline Reinforcement Learning for Bandwidth Estimation in Real Time Communications<\/p>\n\n\n\n\n\n\n
Video conferencing systems have recently emerged as indispensable tools to sustain global business operations and enable accessible education by revolutionizing the way people connect, collaborate, and communicate despite physical barriers and geographical divides. The quality of experience (QoE) delivered by these systems to the end user depends on bandwidth estimation<\/em>, which is the problem of estimating the variable capacity of the bottleneck link between the sender and the receiver over time. In real time communication systems (RTC), the bandwidth estimate serves as a target bit rate for the audio\/video encoder, controlling the send rate from the client. Overestimating the capacity of the bottleneck link causes network congestion as the client sends data at a rate higher than what the network can handle. Network congestion is characterized by increased delays in packet delivery, jitter, and potential packet losses. In terms of user\u2019s experience, users will typically experience many resolution switches, frequent video freezes, garbled speech, and audio\/video desynchronization, to name a few. Underestimating the available bandwidth on the other hand causes the client to encode and transmit the audio\/video streams in a lower rate signal than what the network can handle, which leads to underutilization and degraded QoE. Estimating the available bandwidth accurately is therefore critical to providing the best possible QoE to users in RTC systems. Nonetheless, bandwidth estimation is faced with a multitude of challenges such as dynamic network paths between senders and receivers with fluctuating traffic loads, existence of diverse wired and wireless access network technologies with distinct characteristics, existence of different transmission protocols fighting for bandwidth to carry side and cross traffic, and partial observability of the network as only local packet statistics are available at the client side to base the estimate on. <\/p>\n\n\n\n To improve QoE for users in RTC systems, the ACM MMSys 2024 grand challenge focuses on the development of a deep learning-based bandwidth estimator using offline reinforcement learning (RL) techniques. A real-world dataset of observed network dynamics with objective metrics that reflect user-perceived audio\/video quality in Microsoft Teams is released to train the deep RL policies for bandwidth estimation. <\/p>\n\n\n\n Please NOTE<\/strong> that the intellectual property (IP) is not transferred to the challenge organizers, i.e., participants remain the owners of their code (when the code is made publicly available, an appropriate license should be added).<\/em><\/p>\n\n\n\n Offline RL is a variant of RL where the agent learns from a fixed dataset of previously collected experiences, without interacting with the environment during training. In offline RL, the goal is to learn a policy that maximizes the expected cumulative reward based on the dataset. Offline RL is different from online RL where the agent can interact with the environment using its updated policy and learn from the feedback it receives online. <\/p>\n\n\n\n In this challenge, participants are provided with a dataset of real-world trajectories for Microsoft Teams audio\/video calls. Each trajectory corresponds to the sequence of high-dimensional observation vector (on<\/sub><\/em>) computed based on packet information received by the client in one audio\/video call, along with the bandwidth estimates (bn<\/sub><\/em>) predicted by different estimators (behavior policies). Objective signals which capture the user-perceived audio\/video quality during the call are provided. These objective signals are predicted by ML models whose predictions have high correlation with subjective audio and video quality scores as determined by ITU-T’s P.808 and P.910, respectively.<\/p>\n\n\n\n The goal of the challenge is to improve QoE for RTC system users as measured by objective audio\/video quality scores by developing a deep learning-based policy model (receiver-side bandwidth estimator, \u03c0) with offline RL techniques, such as conservative Q-learning, inverse reinforcement learning, and constrained policy optimization, to name a few. To this end, participants are free to specify an appropriate reward function based on the provided dataset of observed network dynamics and objective metrics, the model architecture, and the training algorithm, given that the developed model adheres to the below requirements.<\/p>\n\n\n\n Failing to adhere to challenge rules will lead to disqualification from the challenge.<\/p>\n\n\n\n Evaluation of submitted models will be conducted in a 2-stage evaluation process: <\/p>\n\n\n\n In either evaluation stage, the following scoring function will be used to rank the models: <\/p>\n\n\n\n Ecalls<\/sub>[En<\/sub>[objective video quality + objective audio quality]]<\/em><\/strong><\/p>\n\n\n\n The outer expectation is across all calls and across repetitions per call, while the inner expectation is the temporal average of the objective metrics throughout a call.<\/p>\n\n\n\n There are two steps in registering for the challenge: <\/p>\n\n\n\n Organizers will communicate updates regarding the grand challenge and announce the availability of data, and evaluation results etc. via email. Challenge leaderboard for each evaluation stage will be posted on the challenge website.<\/strong><\/p>\n\n\n\n If accepted, paper submissions to the Open-source Software and Datasets (opens in new tab)<\/span><\/a> will be included in the ACM MMSys 2024 conference proceedings, and code submissions will be given the appropriate reproducibility badge (opens in new tab)<\/span><\/a>. Authors of accepted papers will also have a chance to present their work during the conference. Moreover, the winner and the runner-up, based on the results of the second evaluation stage<\/strong>, will be awarded cash prizes as described in the rules section<\/a>.<\/p>\n\n\n\n Participants with queries related to this grand challenge can either contact Sami Khairy by email<\/a> or create an issue on the Github repository.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n\n\n SPONSOR<\/p>\n\n\n\n These Official Rules (\u201cRules\u201d) govern the operation of the ACM MMSys 2024 Bandwidth Estimation Challenge Event Contest (\u201cContest\u201d). Microsoft Corporation, One Microsoft Way, Redmond, WA, 98052, USA, is the Contest sponsor (\u201cSponsor\u201d).<\/p>\n\n\n\n DEFINITIONS<\/p>\n\n\n\n In these Rules, \u201cMicrosoft\u201d, \u201cwe\u201d, \u201cour\u201d, and \u201cus\u201d, refer to Sponsor, and \u201cyou\u201d and \u201cyourself\u201d refers to a Contest participant or the parent\/legal guardian of any Contest entrant who has not reached the age of majority to contractually obligate themselves in their legal place of residence. \u201cEvent\u201d refers to the ACM MMSys 2024 Bandwidth Estimation Challenge event held in Bari, Italy (the \u201cEvent\u201d). By entering you (your parent\/legal guardian if you are not the age of majority in your legal place of residence) agree to be bound by these Rules.<\/p>\n\n\n\n ENTRY PERIOD<\/p>\n\n\n\n The Contest will operate from October 9, 2023 to April 18, 2024 (\u201cEntry Period\u201d). The Entry Period is divided into several periods as described in Section 5 How to Enter.<\/p>\n\n\n\n ELIGIBILITY<\/p>\n\n\n\n Open to any registered Event attendee 18 years of age or older. If you are 18 years of age or older but have not reached the age of majority in your legal place of residence, then you must have the consent of a parent\/legal guardian. Employees and directors of Microsoft Corporation and its subsidiaries, affiliates, advertising agencies, and Contest Parties are not eligible, nor are persons involved in the execution or administration of this promotion, or the family members of each above (parents, children, siblings, spouse\/domestic partners, or individuals residing in the same household). Void in Cuba, Iran, North Korea, Sudan, Syria, Region of Crimea, and where prohibited. For business\/tradeshow events: If you are attending the Event in your capacity as an employee, it is your sole responsibility to comply with your employer\u2019s gift policies. Microsoft will not be a party to any disputes or actions related to this matter. PLEASE NOTE: If you are a public sector employee (government and education), all prize awards will be awarded directly to your public sector organization and subject to receipt of a gift letter signed by your agency\/institution\u2019s ethics officer, attorney, or designated executive\/officer responsible for your organization\u2019s gifts\/ethics policy. Microsoft seeks to ensure that by offering items of value at no charge in promotional settings it does not create any violation of the letter or spirit of the entrant\u2019s applicable gifts and ethics rules.<\/p>\n\n\n\n HOW TO ENTER<\/p>\n\n\n\n The Contest Objective is to improve QoE for video conferencing system users as measured by objective audio\/video quality scores by developing a deep learning-based model (receiver-side bandwidth estimator, \u03c0) with offline RL techniques. To this end, participants should specify an appropriate reward function based on the provided dataset, the model architecture, and the training algorithm, given that the developed model adheres to the challenge requirements described in the homepage. Prizes will be awarded to the top 2 performing models based on the evaluation criteria and methodology described in the homepage. You may participate as an individual or a team. If forming a team, you must designate a \u201cTeam Captain\u201d who will submit all entry materials on behalf of the team. Once you register as part of a Team, you cannot change Teams or alter your current team (either by adding or removing members) after the submission of your Entry. Limit one Entry per person and per team. You may not compete on multiple teams, and you may not enter individually and on a team. We are not responsible for Entries that we do not receive for any reason, or for Entries that we receive but are not decipherable or not functional for any reason. Each Team is solely responsible for its own cooperation and teamwork. In no event will Microsoft officiate in any dispute regarding the conduct or cooperation of any Team or its members. The Contest will operate as follows:<\/p>\n\n\n\n Registration \/ Development Period: October 9, 2023 \u2013 January 5, 2024<\/strong><\/p>\n\n\n\n There are two steps in registering for the challenge: <\/p>\n\n\n\n Organizers will communicate updates regarding the grand challenge and announce the availability of data, and evaluation results etc. via email. Challenge leadership board and final results will be posted on the challenge website.<\/p>\n\n\n\n After registration, participants will develop a bandwidth estimation model that best meets the contest objective and the challenge requirements as described in the homepage.<\/p>\n\n\n\n The final evaluation will be conducted in a 2-stage process as described in the homepage during the judging period: January 27, 2024, 12:00 AM AoE\u2013 March 1, 2024, 11:59 PM AoE. <\/p>\n\n\n\n To submit your entry, submit your model, code, and paper through the grand challenge conference submission system. You must satisfy all the challenge requirements as described on the homepage. ACM MMSys 2024 grand challenge submissions are due on January 5, 2024, 11:59 PM AoE. The entry limit is one per person during the Entry Period. Any attempt by any you to obtain more than the stated number of entries by using multiple\/different accounts, identities, registrations, logins, or any other methods will void your entries and you may be disqualified. Use of any automated system to participate is prohibited.<\/p>\n\n\n\n We are not responsible for excess, lost, late, or incomplete entries. If disputed, entries will be deemed submitted by the \u201cauthorized account holder\u201d of the email address, social media account, or other method used to enter. The \u201cauthorized account holder\u201d is the natural person assigned to an email address by internet or online service provider, or other organization responsible for assigning email addresses.<\/p>\n\n\n\n PAPER FORMAT<\/p>\n\n\n\n Accepted challenge papers are published in the proceedings of the ACM MMSys 2024. Paper format should adhere to ACM’s author guidelines (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n ELIGIBLE ENTRY<\/p>\n\n\n\n To be eligible, an entry must meet the following content\/technical requirements:<\/p>\n\n\n\n USE OF ENTRIES<\/p>\n\n\n\n We are not claiming ownership rights to your Submission. However, by submitting an entry, you grant us an irrevocable, royalty-free, worldwide right and license to use, review, assess, test, and otherwise analyze your entry and all its content in connection with this Contest and use your entry in any media whatsoever now known or later invented for any non-commercial or commercial purpose, including, but not limited to, the marketing, sale or promotion of Microsoft products or services, without further permission from you. You will not receive any compensation or credit for use of your entry, other than what is described in these Official Rules.<\/p>\n\n\n\n By entering you acknowledge that we may have developed or commissioned materials similar or identical to your entry and you waive any claims resulting from any similarities to your entry. Further, you understand that we will not restrict work assignments of representatives who have had access to your entry and you agree that the use of information in our representatives\u2019 unaided memories in the development or deployment of our products or services does not create liability for us under this agreement or copyright or trade secret law.<\/p>\n\n\n\n Your entry may be posted on a public website. We are not responsible for any unauthorized use of your entry by visitors to this website. We are not obligated to use your entry for any purpose, even if it has been selected as a winning entry.<\/p>\n\n\n\n WINNER SELECTION AND NOTIFICATION<\/p>\n\n\n\n Pending confirmation of eligibility, potential prize winners will be selected by Microsoft or their Agent or a qualified judging panel from among all eligible entries received based on the following 2-stage judging process: <\/p>\n\n\n\n In either judging stage, the following scoring function will be used to rank the models: <\/p>\n\n\n\n Ecalls<\/sub>[En<\/sub>[objective video quality + objective audio quality]]<\/em><\/strong><\/p>\n\n\n\n The outer expectation is across all calls and across repetitions per call, while the inner expectation is the temporal average of the objective metrics throughout a call.<\/p>\n\n\n\n This challenge metric gives an equal weighting between objective audio and video quality. Among the submitted proposals, if the difference between overall evaluation metric between the models is not statistically significant, the model with a lower number of operations per second will be given a higher ranking. Winners will be notified after the end of the judging period on March 1, 2024.<\/p>\n\n\n\n In the event of a tie between any eligible entries, an additional judge will break the tie based on the judging criteria described above. The decisions of the judges are final and binding. If we do not receive enough entries meeting the entry requirements, we may, at our discretion, select fewer winners. If public vote determines winners, it is prohibited for any person to obtain votes by any fraudulent or inappropriate means, including offering prizes or other inducements in exchange for votes, automated programs, or fraudulent IDs. Microsoft will void any questionable votes.<\/p>\n\n\n\n PRIZES<\/p>\n\n\n\n The following prizes will be awarded:<\/p>\n\n\n\n First Prize: the winner will receive: $2000, paid in the form of a check from the sponsor or donation to the Individual Entrant or Team Captain\u2019s Educational institution. Approximate Retail Value (ARV) $2,000.00. All prizes are stated in US dollars.<\/p>\n\n\n\n The total Approximate Retail Value (ARV) of all prizes: $3,000.00.<\/p>\n\n\n\n PLEASE NOTE: If you are a public sector employee (government and education), your prize will be awarded directly to your public sector organization as a donation, and subject to receipt of a gift letter signed by your agency\/institution\u2019s ethics officer, attorney, or designated executive\/officer responsible for your organization\u2019s gifts\/ethics policy.<\/mark><\/p>\n\n\n\n We will only award one prize per person. No more than the stated number of prizes will be awarded. No substitution, transfer, or assignment of prize permitted, except that Microsoft reserves the right to substitute a prize of equal or greater value in the event the offered prize is unavailable. Prizes are awarded \u201cAS IS\u201d with no warranty of any kind, either express or implied, including but not limited to, the implied warranties or merchantability, fitness for a particular purpose, or non-infringement. Prize winners may be required to complete and return prize claim and \/ or tax forms (\u201cForms\u201d) within the deadline stated in the winner notification. Taxes on the prize, if any, are the sole responsibility of the winner, who is advised to seek independent counsel regarding the tax implications of accepting a prize. By accepting a prize, you agree that Microsoft may use your entry, name, image and hometown online and in print, or in any other media, in connection with this Contest without payment or compensation to you, except where prohibited by law.<\/p>\n\n\n\n ODDS<\/p>\n\n\n\n The odds of winning are based on the number and quality of eligible entries received.<\/p>\n\n\n\n GENERAL CONDITIONS AND RELEASE OF LIABILITY<\/p>\n\n\n\n To the extent allowed by law, by entering you agree to release and hold harmless Microsoft and its respective parents, partners, subsidiaries, affiliates, employees, and agents from any and all liability or any injury, loss, or damage of any kind arising in connection with this Contest or any prize won.<\/p>\n\n\n\n All local laws apply. The decisions of Microsoft are final and binding.<\/p>\n\n\n\n We reserve the right to cancel, change, or suspend this Contest for any reason, including cheating, technology failure, catastrophe, war, or any other unforeseen or unexpected event that affects the integrity of this Contest, whether human or mechanical. If the integrity of the Contest cannot be restored, we may select winners from among all eligible entries received before we had to cancel, change or suspend the Contest.<\/p>\n\n\n\n If you attempt or we have strong reason to believe that you have compromised the integrity or the legitimate operation of this Contest by cheating, hacking, creating a bot or other automated program, or by committing fraud in any way, we may seek damages from you to the full extent of the law and you may be banned from participation in future Microsoft promotions.<\/p>\n\n\n\n GOVERNING LAW<\/p>\n\n\n\n This Contest will be governed by the laws of the State of Washington, and you consent to the exclusive jurisdiction and venue of the courts of the State of Washington for any disputes arising out of this Contest.<\/p>\n\n\n\n PRIVACY<\/p>\n\n\n\n At Microsoft, we are committed to protecting your privacy. Microsoft uses the information you provide on this form to notify you of important information about our products, upgrades and enhancements, and to send you information about other Microsoft products and services. Microsoft will not share the information you provide with third parties without your permission except where necessary to complete the services or transactions you have requested, or as required by law. Microsoft is committed to protecting the security of your personal information. We use a variety of security technologies and procedures to help protect your personal information from unauthorized access, use, or disclosure. Your personal information is never shared outside the company without your permission, except under conditions explained above.<\/p>\n\n\n\n If you believe that Microsoft has not adhered to this statement, please contact Microsoft by sending an email to\u202fprivrc@microsoft.com<\/a>\u202for postal mail to Microsoft Privacy Response Center, Microsoft Corporation, One Microsoft Way, Redmond, WA.<\/p>\n\n\n\n\n\n We will release a dataset of trajectories for Microsoft Teams audio\/video calls along with two objective signals that measure user-perceived audio\/video quality in audio\/video calls. The data is collected from audio\/video peer-to-peer Microsoft Teams calls conducted between testbed nodes that are geographically distributed across many countries and continents. Nodes are connected to the internet through various Internet Service Providers (ISPs) using either wired or wireless connections. Calls have been conducted with different bandwidth estimators, i.e., behavior policies, including traditional methods such as Kalman-filtering-based estimators and WebRTC (Web Real Time Communications), as well as different ML policies. <\/p>\n\n\n\n Each trajectory corresponds to one audio\/video call leg and consists of a sequence of:<\/p>\n\n\n\n The observation vector at a time step n<\/em> encapsulates observed network statistics that characterize the state of the bottleneck link between the sender and receiver over the 5<\/strong> most recent short term monitor intervals (MI) of 60ms<\/em> and the 5<\/strong> most recent long-term MIs of 600ms<\/em>. Specifically, the observation vector tracks 15<\/strong> different network features over 5<\/strong> short and 5<\/strong> long term MIs (15 features x (5 short term MIs + 5 long term MIs) = 150)<\/strong>. The 15 features and their description are as follows. Features are based on packets received during the short and long term monitor intervals.<\/p>\n\n\n\n The indices (zero-indexed) of features over the 5 short term MIs are {(feature number – 1)*10, ., ., ., feature number * 10 – 5 – 1}<\/strong>.<\/p>\n\n\n\n The indices (zero-indexed) of features over the 5 long term MIs are {feature number * 10 – 5, ., ., ., feature number * 10 – 1}<\/strong>.<\/p>\n\n\n\n We are going to release data from 18859<\/strong> calls conducted between testbed nodes to be used as training and validation data by the participants. Participants are free to split the data into train\/validation sets as deemed necessary. <\/p>\n\n\n\n We will also release a dataset from 9405<\/strong> emulated test calls which contains ground truth information about the bottleneck link between the sender and receiver, namely, bottleneck capacity and loss rate, in addition to the aforementioned data. In this dataset, the characteristics of the bottleneck, namely ground truth capacity and loss rate, are randomly varied throughout the duration of the test call to generate a diverse set of trajectories with network dynamics that may not occur in the real world but are nevertheless important to enhance state-action space coverage and aid in learning generalizable policies.<\/p>\n\n\n\n Participants are free to use this dataset in conjunction with the real-world testbed dataset to train the policy, given that ground truth information is not provided as input to the model and the true environment remains partially observable (only through the observation vector).<\/strong> Any model which has ground truth information in the inputs will be disqualified from the contest.<\/strong><\/p>\n\n\n\n\n\n Time zone for below dates is Anywhere on Earth<\/em> (AoE)<\/p>\n\n\n\nChallenge task <\/h3>\n\n\n\n
Challenge requirements<\/h3>\n\n\n\n
\n
Evaluation criteria and methodology <\/h3>\n\n\n\n
\n
Registration procedure<\/h3>\n\n\n\n
\n
Awards<\/h3>\n\n\n\n
Contact us<\/h3>\n\n\n\n
Official rules<\/h2>\n\n\n\n
\n
\n
\n
Second Prize: the runner-up will receive: $1000, paid in the form of a check from the sponsor or donation to the Individual Entrant or Team Captain\u2019s Educational Institution. Approximate Retail Value (ARV) $1,000.00.<\/p>\n\n\n\nDataset Description<\/h2>\n\n\n\n
\n
\n
Training and Validation Datasets <\/h3>\n\n\n\n
Emulated Dataset <\/h3>\n\n\n\n