Azure Monitor alerts help Microsoft pay billions of dollars of bills on time

Jul 25, 2019   |  

With billions of dollars in payments processed annually, punctuality is imperative.

Microsoft pays suppliers and partners for a variety of services and products via thousands of transactions each month; accurate and on-time payment is crucial to its relationships and success. The sheer volume makes mistakes inevitable, but the Finance Team within Microsoft Digital is working to at least make them invisible.

“We didn’t want to rely on our partners or stakeholders to find problems with payments,” says Ullas Kumble, lead software engineer in Microsoft Digital.

Over the past six months, the payments team set out to create a platform that would address the challenge of missed and late payments. SAP is the core processing engine for payments, and integration would be needed to identify issues throughout the process. The team began by creating events triggered by processing delays or application errors, defined by service-level agreements (SLAs) and business rules.

“The payment process starts with an invoice, either in SAP, or an upstream system integrated with SAP. Once approved for payment, the payment is initiated through one of our banking partners,” says Kumble. “Our need for customized procurement and payment tools integrated with SAP provides experiences tailored for our business, but also complexity.”

[Learn how another team inside Microsoft Digital is designing a modern service for the cloud or how Microsoft has automated a SOX control using AI.]

Microsoft uses a hybrid SAP implementation, according to the team—with SAP being the business process and transaction engine, and a multitude of other applications acting as the user interface. This allows great freedom to integrate business processes into existing applications but challenges in monitoring end-to-end processes.

“We identified the SAP and upstream system data our stakeholders and support engineers needed to understand and act on an issue,” says Loviee Makkar, a Microsoft Digital software engineer. “The challenge was stitching the data together in a way that would help both our business stakeholders and operations teams.”

The team investigated services in Microsoft Azure that would achieve the goal, and found Azure Monitor Alerts to almost perfectly fit its needs.

“The payments team was one of the early adopters and acted as a great partner for Azure Monitor alerts,” says Vijay Nagarajan, senior program manager on the Azure Monitor alerts team, “providing feedback throughout and working with the alerts team to effectively utilize the functionality.”

“The simplicity of the solution is by just adding alerts to our Azure Monitor Application Insights instance we could both detect and notify our teams of issues as they occurred,” says Makkar.

Makkar expands, “We query SAP business process status through the Unified Telemetry Platform (UTP) and correlate that with our own telemetry from payments services. This enables us to detect issues across both systems and send notifications.”

Although Azure alerts provide built-in capabilities for notifications, the team decided to customize the implementation for different audiences and integration into support systems.

“Our business stakeholders need more context for the alerts. They are interested in the SLA for the transaction, type of payment, and status of the transaction,” says Makkar. “We have to translate often cryptic or complex errors into common language with only the relevant details included.”

Image shows the payment alerts framework hosted on Azure. From left, SAP and payments services application send data to Application insights. Application insights sends data to Logic App when threshold is met, outlook connector is initiated to send emails to DRI and business team.
The payments alert framework uses alerts from Application Insights to trigger workflows in Azure Logic Apps that create custom emails for different audiences and ticket generation.

The team uses the alerts service to integrate with Azure Logic Apps. This provides the ability to create custom email templates and to integrate with their ticketing system.

“Logic apps provides a serverless architecture that can scale as needed and allow us to highly customize actions for an alert based on our definition,” says Makkar. “We are able to create a simple table using Azure Functions to combine errors and provide relevant details to the different stakeholders.”

The service has already improved the payment process and results.

“We have improved DRI efficiency by 30 to 40 percent since implementing the payments alert framework, resolving issues sooner and requiring far less work from engineers to investigate data,” says Makkar.

The team’s work also helped to improve the service for Microsoft customers. “The payments team’s continued collaboration with Azure Monitor to create consolidating telemetry across apps and solutions as well as mechanisms for alerting has created refined experiences,” says Nagarajan.

Kumble says, “We are just starting. Our long-term focus is on machine learning—using the large amount of data to identify transactions that are likely to fail or anomalies in the payment process. We want to have a safe, secure, and accurate payment service that supports Microsoft’s growing business.”

Learn how another team inside Microsoft Digital is designing a modern service for the cloud or how Microsoft has automated a SOX control using AI.