{"id":8039,"date":"2023-11-20T01:31:38","date_gmt":"2023-11-20T09:31:38","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=8039"},"modified":"2023-11-20T11:19:48","modified_gmt":"2023-11-20T19:19:48","slug":"listening-to-millions-of-iot-building-sensor-readings-with-load-testing","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/listening-to-millions-of-iot-building-sensor-readings-with-load-testing\/","title":{"rendered":"Listening to millions of IoT building sensor readings with load testing"},"content":{"rendered":"

\"MicrosoftAs an employee, an easy-seeming task like booking a meeting room can quickly become difficult if you cannot see which rooms are available\u2014that is where taking a transformed approach to load testing can help.<\/p>\n

Employees booking rooms and dozens of other similar activities in Microsoft buildings generate significant Internet of Things (IoT) sensor data. All these spatial data readings must be processed in near real-time so that the information remains relevant and significant. If you are looking or searching for a meeting room and cannot figure out whether it is occupied or not within seconds, not only will you walk out the door but also waste your time looking for other rooms, losing productivity. And if the IoT motion sensor data isn\u2019t processed as fast as the employee is entering or leaving the room, you are left with incorrect and outdated information. That lag in information retrieval\u2014even just a few milliseconds\u2014can significantly degrade an employee\u2019s experience.<\/p>\n

The way to power employee experiences is to understand the data behind occupancy, room temperatures, hot desking, and more. The question then becomes, \u2018How do you power all these employee experiences that are a part of your workplace?\u2019<\/p>\n

\u2014Apoorv Gupta, senior software engineer, Microsoft Digital Employee Experience<\/p>\n<\/blockquote>\n

Sensors in Microsoft buildings take more than two million IoT readings per day, with an expected traffic growth of around 30 million readings per day.<\/p>\n

\u201cThe way to power employee experiences is to understand the data behind occupancy, room temperatures, hot desking, and more,\u201d says Apoorv Gupta, a senior software engineer for Microsoft Digital Employee Experience, the team building a new load testing service for the Microsoft Global Workplace Services (GWS) team that manages the company\u2019s buildings. \u201cThe question then becomes, \u2018How do you power all of these employee experiences that are a part of your workplace?\u2019\u201d<\/p>\n

Microsoft\u2019s solution for tracking all these sensor readings was to deploy a Digital Integration Platform that brings all those people, places, and devices together in new and better ways. This solution can simultaneously process data from multiple employee experiences in over 300 buildings across the globe\u2014without compromising system stability, reliability, and load performance. Microsoft Azure Digital Twins powers the Digital Integration Platform on the backend, storing all the sensor data and using it for processing. To load test the solution the team decided to build a full-blown load testing solution leveraging open-source technologies.<\/p>\n

I challenged the team to prove that our Digital Integration Platform can scale to integrate with a host of new sensor types, and scale to enable real-time IoT-powered experiences for our global workforce. The MS Digital team met the challenge, leveraging configurable cloud automation to validate the platform at scale, for a remarkably low cost.<\/p>\n

\u2014Emmanuel Daniel, director of Digital Transformation and Smart Buildings, Global Workplace Services<\/p>\n<\/blockquote>\n

\u201cAzure Digital Twins makes it easy to merge the physical and digital worlds by allowing us to map data to device to building,\u201d says Emmanuel Daniel, director of Digital Transformation and Smart Buildings for GWS. \u201cThis enables our employees to have great experiences in the physical space.\u201d<\/p>\n

According to Daniel, employee experiences are dependent on multiple data points, from many sensors producing a large volume of messages. And for the experiences to successfully work, the platform needs to scale. For example, the size and scale of Microsoft\u2019s Puget Sound-area East Campus makes it very difficult to mirror the environment.<\/p>\n

\u201cI challenged the team to prove that our Digital Integration Platform can scale to integrate with a host of new sensor types, and scale to enable real-time IoT-powered experiences for our global workforce,\u201d Daniel says. \u201cThe Microsoft Digital team met the challenge, leveraging configurable cloud automation to validate the platform at scale, for a remarkably low cost.\u201d<\/p>\n

[Check out inside Microsoft\u2019s smart building in Herzliya, Israel<\/a>. Read more about accelerating Microsoft\u2019s global real estate transformation with Microsoft Digital<\/a>. Discover how Microsoft operations evolve with Azure<\/a>. Look into building Microsoft’s employee-centric experience<\/a>.]<\/em><\/p>\n

Leveraging open-source technology for faster deployment<\/h2>\n

To start the testing process, the team gathered three months of baseline load data from GWS\u2019 previous tool and used that as a benchmark. Then the team created tests that measured how well the system could handle the number of readings\/requests per second (RPS) received and overall load distribution. For example, if you have a continuous stream of sensor readings being ingested across hundreds of buildings all at the same time on a global scale, that is a massive load of requests that need to be processed in near real-time between all the types of IoT data that is being processed.<\/p>\n

The team ran a complete cycle of load tests, including X load (peak load), twice the peak (2X), and a long-haul endurance run that tracked performance over 24-48 hours. These tests are crucial to ensure that the components as well as the entire system can function at that high-intensity level. They also prepare the system to handle higher loads as additional employee use cases continue to be added.<\/p>\n

Once you test the system, you need to determine whether it behaves how you want it to, and how it behaves under load. You need to generate the load and simulate it to mimic the real-life scenario.<\/p>\n

\u2014Mridul Verma, senior software engineer, Microsoft Digital Employee Experience<\/p>\n<\/blockquote>\n

\u201cWe already knew our solution was going to need to support millions of readings in the future, so we needed to be ready for that by making sure it\u2019s stable under that load,\u201d says Shuvajyoti Deb, a software engineer for the Microsoft Digital Employee Experience team. \u201cWe helped create the methodology and implementation plan that we used to deploy the load testing solution.\u201d<\/p>\n

\"Deb,
Shuvajyoti Deb (left), Amit Agrawal (center), and Apoorv Gupta helped create the methodology and procedures for the load testing solution Microsoft is using to track millions of IoT sensor readings. Agrawal and Deb are software engineers and Gupta is a senior software engineer on the Microsoft Digital Employee Experience team. (Photos by Shuvajyoti Deb, Amit Agrawal, and Apoorv Gupta)<\/figcaption><\/figure>\n

The team opted to use Apache JMeter and Microsoft Azure Kubernetes Service (AKS) clusters to run the load testing solution. The load testing solution mimics the people, places, devices, sensor types and their locations in all the buildings so the team can see how the Digital Integration Platform would operate in real life.<\/p>\n

\u201cOnce you test the system, you need to determine whether it behaves how you want it to, and how it behaves under load. You need to generate the load and simulate it to mimic the real-life scenario,\u201d says Mridul Verma, a senior software engineer on the Microsoft Digital Employee Experience team who helped set load testing benchmarks and identified bottlenecks.<\/p>\n

Throughout the course of testing, the team focused on various metrics related to reliability, performance, and availability to ensure that the solution met the benchmarks and expectations that were set at the beginning of the project. Upon completing each round of testing, the team ran an analysis and determined where it could alter the instances or configurations to better optimize the system. Once the alterations were made the team ran the same set of tests again to see if performance improved and continued to optimize until it met expectations. \u00ad\u00ad<\/p>\n

The last stage of the project was creating automated pipelines and making them available on GitHub so anyone in GWS can perform load testing. The availability of the solution on GitHub makes it a lot easier and faster for teams to conduct the load testing for their respective services and platforms. Additionally, the availability of the load testing solution on GitHub makes it open to customization as required.<\/p>\n

\u201cGitHub makes it easy for users to access the load testing solution quickly contribute to it to meet their needs,\u201d says Amit Agrawal, another software engineer on the Microsoft Digital Employee Experience team. His role was to develop the methodology and process for load testing.<\/p>\n

Optimizing today for future growth tomorrow<\/h2>\n

Load testing helps Microsoft gain insight into the IoT sensor data it collects to give employees a better experience in its buildings. With a more stable and reliable platform, more employee experiences can be added, more facilities can be onboarded, and more teams can use this tool to make data quickly available. As a result, employees are more productive in their day-to-day tasks, saving them time during the week.<\/p>\n

Looking forward, the team will onboard the company’s new Eastside Campus in the Puget Sound region to the Digital Integration Platform, where the company will be opening 17 new smart buildings starting in late 2023.<\/p>\n

\"Key<\/p>\n