{"id":18640,"date":"2025-03-06T09:00:00","date_gmt":"2025-03-06T17:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=18640"},"modified":"2026-03-24T13:26:00","modified_gmt":"2026-03-24T20:26:00","slug":"boosting-network-performance-at-microsoft-with-real-time-telemetry-and-performance-visualization","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/boosting-network-performance-at-microsoft-with-real-time-telemetry-and-performance-visualization\/","title":{"rendered":"Boosting network performance at Microsoft with real-time telemetry and performance visualization"},"content":{"rendered":"\n

For the network engineers who keep Microsoft connected across thousands of devices at more than 800 sites around the world, time is always of the essence. They need to monitor network performance, identify outages, and keep devices running smoothly. Otherwise, we risk severe productivity setbacks and security risks.<\/p>\n\n\n\n

\"Microsoft<\/figure>\n\n\n\n

Our team in Microsoft Digital, the company\u2019s IT organization, has been enabling up-to-date data for a growing number of network devices through real-time streaming telemetry. Now, we\u2019ve created a powerful new tool that provides greater visibility than ever before.<\/p>\n\n\n\n

The Live Streaming Panel is a new feature for Infrastructure Graph (IGraph), an internal Microsoft tool that maps the topology of our enterprise network. Through real-time visual mapping and low-latency metrics, it unlocks new capabilities for network engineers, helping keep our company connected and productive.<\/p>\n\n\n\n

If you\u2019re planning on providing your network engineers with more easily visualized, more up-to-date data, the lessons we learned while creating this solution can help guide your work.<\/p>\n\n\n\n

The urgency of real-time telemetry for network devices<\/h2>\n\n\n\n
\"Sinha,
Astha Sinha (left to right), Vinod Kumar Singh, Faiz Gouri, and Nevedita Mallick are part of the team that created the Live Metrics Panel, and Manjiri Keskar<\/em> (far right) leads a team of network engineers who benefit from this new tool.<\/em><\/figcaption><\/figure>\n\n\n\n

Our network engineers deploy and maintain the network devices that keep Microsoft running. They serve as direct response individuals (DRIs), handling reports of outages as part of our incident management (ICM) process.<\/p>\n\n\n\n

Imagine a network incident that leaves an entire corporate facility with little or no connectivity, preventing potentially hundreds of employees from doing their work. Worse, consider an outage affecting an app or solution Microsoft customers use. Resolving these kinds of issues is mission-critical, and every second counts.<\/p>\n\n\n\n

Many legacy network devices are only compatible with the Simple Network Management Protocol (SNMP), which retrieves log data at lengthy intervals stretching anywhere from five minutes to six hours. This lag means slower mean time to detection and mean time to resolution for engineers tasked with solving network issues.<\/p>\n\n\n\n

\n

\u201cWe wanted a more proactive approach to monitoring and data observability, which aligns with our goal to increase the reliability of network infrastructure at Microsoft and strengthen our overall security posture,\u201d says Astha Sinha, senior product manager on our Real-Time Telemetry team in Microsoft Digital.<\/p>\n<\/blockquote>\n\n\n\n

Modern network devices using the gRPC Network Management Interface (gNMI) telemetry protocol operate on a push-based model, enabling streaming telemetry at intervals as short as one minute. Implementing streaming telemetry at Microsoft has been transformative, allowing network engineers to access near real-time data for any onboarded devices so they can find and fix outages in minutes\u2014not hours<\/a>.<\/p>\n\n\n\n

The next step was layering intuitive observability on top of that technology. Instead of signing in to multiple tools and constructing Kusto queries in Azure to investigate issues, engineers would benefit from simple, visual data access.<\/p>\n\n\n\n

\n

\u201cWith just a raw data interface, network engineers can certainly find issues, but they want to see data over time, in real time,\u201d says Faiz Gouri, a senior software engineer on the Microsoft IGraph team. \u201cIt becomes possible to identify trends when the history is more visual.\u201d<\/p>\n<\/blockquote>\n\n\n\n

This approach led to the creation of the Live Streaming Panel for IGraph, modernizing our network with real-time insights, proactive monitoring, enhanced network management capabilities, and faster incident resolution.<\/p>\n\n\n\n

Unlock instant visibility into network device performance<\/h2>\n\n\n\n

IGraph offers a visual platform to correlate vital metrics across network devices, helping our network engineers achieve a truly intelligent infrastructure. The Live Streaming Panel, its latest feature, lets teams select devices and interfaces to view critical metrics that include throughput, utilization, packet drops, and errors. It displays historical data first, followed by live updates for devices capable of real-time telemetry on a minute-by-minute basis.<\/p>\n\n\n\n

To create this feature, our Real-Time Telemetry and IGraph teams collaborated with network engineers from different service lines, including members of the WAN, wired, wireless, and cloud teams. We identified pain points that included stale log data, difficulties isolating issues, and the need to sign in to network devices to access their information, which consumes critical CPU usage and introduces security liabilities.<\/p>\n\n\n\n

We also uncovered core needs for what metrics would be most relevant and the best ways to present that data intuitively. Most of all, network engineers wanted access to free-flowing data on metrics that affect quality of service, without a cumbersome sign-in process or hopping from pane to pane. Initial core metrics included throughput, utilization, and whether links were active or inactive.<\/p>\n\n\n\n

\n

\u201cThe live stream empowers network engineers to monitor critical metrics continuously, enabling them to swiftly identify and address potential issues to enhance overall operational efficiency,\u201d says Vinod Kumar Singh, principal software engineer on the Real-Time Telemetry team. \u201cThis proactive approach not only improves network reliability but also ensures a seamless experience for our users.\u201d<\/p>\n<\/blockquote>\n\n\n\n

The prototype arose out of a Fix, Hack, Learn session, a creative workshop that allows internal teams to create useful passion projects. Through iterative proofs of concept and agile feedback collection, we developed a tool that matches network engineers\u2019 needs for crucial metrics alongside preferences for accessibility, color coding, and legibility.<\/p>\n\n\n\n

After the concept had proven itself, we had the opportunity to create a full-featured solution and start scaling up. Real-time telemetry and metrics started small, but its capabilities are quickly spreading across more of our enterprise networks.<\/p>\n\n\n\n

Of the 50,000 network devices across Microsoft facilities, between 15,000 and 20,000 constitute wired, physical hardware like routers and switches. We decided to start with these devices, then scale to others as the need and demand arose. And because we onboarded our streaming telemetry capabilities to Azure Kubernetes Service, they\u2019re secure and scalable by default.<\/p>\n\n\n\n

Creating your own Live Streaming Panel<\/h3>\n\n\n\n
\n
\n
\n

1.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Fact-finding: <\/strong><\/p>\n\n\n\n

Give your network engineers the chance to share their needs and pain points, along with the metrics that are most crucial for their work and the best ways to present them. Incorporate these elements into your design.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

2.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Proof of concept: <\/strong><\/strong><\/p>\n\n\n\n

Start small with a limited prototype that onboards a small number of network devices. That will give your network engineers the chance to experience the solution\u2019s possibilities without committing to a full build.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

3.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Iterations and feedback: <\/strong><\/p>\n\n\n\n

An agile, iterative approach ensures your solution evolves to meet your network engineers\u2019 needs. Be open to change as you get closer to a complete tool.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

4.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Privacy and compliance: <\/strong><\/p>\n\n\n\n

Ensure the tool you build complies with data privacy regulations and best practices by providing appropriate access controls. Prioritize compliance with relevant standards and policies.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

5.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Scalability and security: <\/strong><\/p>\n\n\n\n

After you\u2019re past the initial proof of concept and you\u2019re ready to build out a full solution, make sure that your technology stack prioritizes security while providing the flexibility to scale. Otherwise, you won\u2019t be able to extend this technology across your organization.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

6.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Prioritization and expansion:<\/strong><\/strong><\/p>\n\n\n\n

In collaboration with your network teams, discuss which areas will provide the most value. From there, continue to scale. After network engineers experience the benefits that a solution like this provides, they\u2019ll be excited to extend it into their own work.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

7.<\/strong><\/p>\n<\/div>\n\n\n\n

\n
\n

Continuous improvement and innovation: <\/strong><\/strong><\/p>\n\n\n\n

Foster a culture of innovation where network engineers are encouraged to experiment with new ideas and technologies, and plan for regular updates and enhancement to the tool based on user feedback and technological advancements. This will help you update according to the latest market trends and enhance security controls as needed.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

If you\u2019re considering creating a tool like this to support your organization\u2019s network engineers, use these lessons from our internal experience to guide you<\/em>.<\/p>\n\n\n\n

At Microsoft, IGraph visualization operates at three levels of altitude:<\/p>\n\n\n\n