At Microsoft, we’re using process mining to accelerate digital transformation within our organization. With process mining, we’re examining our business processes using digital footprints to visualize how our business operates and uncover challenges and bottlenecks. Process mining enables us to dig deeply into our big data and extract meaningful, actionable insights that we can use to make our business operations run more efficiently.

At Microsoft Digital, we’re using process mining to accelerate digital transformation within our organization. With process mining we’re examining our business processes using source digital footprints (system of records) to visualize those processes and uncover challenges and bottlenecks. Process mining enables us to dig deeply into our transactional logs and extract meaningful, transformative insights that we can use to accelerate digital transformation and streamline business processes.

Exploring process mining

Process mining is an emerging data science field within business process management that uses an organization’s transactional digital footprints to examine their business processes and discover process challenges. Using the metaphor of an x-ray, process mining is an x-ray of business processes as they are exposed through data stored in a variety of digital systems. Process mining intelligent algorithms use this data to monitor process performance of business processes.

Process mining is applicable to any business processes that are mostly (or entirely) repetitive and transactional in nature. The only requirement is having data in the form of transaction steps, and date and time stamps that can be used as a basis for process mining. Most companies that have an IT system supporting their business already have this data.

According to Business Wire, the global process analytics market is expected to grow by more than 50 percent year-over-year in the next 5 years, taking from a 180 million-dollar industry in 2018 to an almost 1.5 billion-dollar industry by 2023.

The challenge for any large organization is to find a single application that can provide end-to-end visibility over these processes. Process mining provides the unique capability of joining the data together and illustrating the connected nature of business processes. The primary goal of process mining is to analyze different applicable data sources and perspectives, and then aggregate the data into information that business leaders can use when making business decisions. This enables business leaders to identify issues and bottlenecks that aren’t obvious, and then act to remedy them.

Driving digital transformation with process mining

Digital transformation at Microsoft is about reimagining how we bring together people, data, and processes to create value for our customers while maintaining a competitive advantage in a digital-first world. Process mining is accelerating digital transformation by providing tools to optimize operations using intelligent processes. This accelerates our business responsiveness, improves service levels, and reduces costs. We want to coordinate people and assets more efficiently, respond to issues in real time, and even solve them preemptively.

Specifically, process mining provides value at various stages of the journey into digital transformation:

• Before digital transformation. Process mining can identify and prioritize potential digital transformation scenarios that will result in cost savings and simulate the scenarios to provide approximate results.
• During digital transformation. You can use process mining to validate steps in your digital transformation. Process mining can test and confirm design principles by simulating potential business scenarios to ensure validity and efficiency.
• After digital transformation. Process mining can provide real-time process monitoring and enable intelligent decision making regarding ongoing process improvements.

Process mining enables us to dig deeply into our big data and extract meaningful, transformative insights that we can then use to accelerate our digital transformation and make our business better.

Using process mining at Microsoft Global Trade

At Microsoft, our challenges are much the same as other organizations. We want more visibility into how our business operates so we can make it better and achieve more. We’re continuing to discover how to use process mining to find our business challenges and move forward with improving and refining the relevant processes. As an example, consider how we’ve implemented process mining at Microsoft Global Trade.

Microsoft Global Trade is part of Microsoft’s export shipment process. Global Trade is responsible for picking, packing, and shipping our hardware products to customers and distribution points around the globe. At Global Trade, we had several questions that we needed answers to including:

• Are the export shipments from our distribution center flowing on time?
• If not, how many shipments are pending for complete documentation?
• At what stage are these shipments pending?
• What is the list of delivery orders that are pending?
• Which countries/regions are they going to?
• How are these orders being shipped?
• Who is the carrier/broker?
• Can we get an alert if a shipment is going to be delayed at any point in the process?

We recognized that by putting process mining to work for Global Trade, we could improve our Global Trade business processes.

Creating direction for process improvement

One of the primary concerns that we had at Global Trade was the time it took between a sales order being placed and the product being delivered to the customer. The end-to-end process involved a lot of moving parts and different parties. We knew there was room for improvement, but we weren’t sure how to identify exactly where those opportunities were. Our first step was to establish business goals based on the primary issues that we had. Our two primary areas of concern were:

• Frequent delays of shipments due to a variety of factors, including customs processing, documentation discrepancies, and inability to consistently identify exactly how or why the shipment was delayed.
• Lack of a single, end-to-end view of order tracking from the moment the order was placed to the moment it was signed for by the customer.

Process mining requires process data

Our next step was to identify and examine all of the transactional and logging data sources that are part of the Global Trade process. We were using a supply chain management system comprised of various shipping carriers, a sales system, and an invoice and payment system. Recognizing that the results of process mining are only as accurate as the data used, we worked to connect the various datasets to create a meaningful and clean data source. We ran a data quality exercise, and then began synthesizing the data from the Global Trade process.

The Global Trade process begins with the sales order initiated at our production facilities and ends with the delivery of products to the customer. A flowchart of the physical process is illustrated in Figure 2. The three major components to this process are SAP, our enterprise resource planning solution (represented by blue boxes), our logistics and shipping partners (represented by green boxes), our inventory management systems (represented by purple boxes) and our invoice tracking and customs systems (represented by orange boxes).

 

Examining Global Trade with process mining

The process mining workflow for Global Trade involved four distinct stages:

1. Gather digital footprints.
2. Apply mining techniques to digital footprints.
3. Visualize business processes in real-time.
4. Infuse with AI and machine learning.

Gathering digital footprints

We collected the underlying process data from all of the applicable data sources and assembled it into digital footprints that defined the important qualities of each process step, including the delivery document number, event type, and a date/time stamp defining when the event occurred.

 

Applying mining techniques to digital footprints

Using the digital footprints as our data source, we mined and synthesized the data using process mining algorithms taken from the open source R library. The output from the process mining techniques provided analysis of the Global Trade process in visual form and revealed important details about process order, timing, and other details not immediately apparent in the underlying digital footprints.

 

The insights from process mining analysis provided a different perspective on the Global Trade process. The data it presented created the opportunity for discussions about how and why our real process differed from the ideal process. It showed us where the process deviated from the ideal, allowed us to identify bottlenecks in the process, and provided a measurement of real process performance.

Visualizing business processes in real time

In our next step, we took the process as it was exposed by process mining techniques and created a visualization for the Global Trade process as it was happening. We used real-time monitoring to immediately begin taking corrective actions in the process as issues occurred. As a result, we were able to resolve problems while the process was ongoing and prevent those problems from causing issues elsewhere in the process.

Visualization was the key to a complete understanding of the real-time Global Trade process. We used JavaScript APIs for Microsoft Visio to embed Visio diagrams into Microsoft SharePoint Online pages for access by anyone who needed to examine the process. Figure 5 represents a real-time visualization of the Global Trade process, with box colors identifying process status as defined by business rules extracted from process mining insights.

 

Infusing with AI and machine learning

At Global Trade, we’re currently looking at ways to predict and prescribe future process scenarios using AI and machine learning. For example, we’ve created a bot to perform interactive question and answer handling through a chat interface. The bot uses Microsoft Cognitive Services natural-language processing to answer natural language questions using data synthesized from processing mining activities. It can respond to a question such as, “How long will it take to complete order number 8611620108?” with a real-time, data-driven response in natural language.

 

How to use process mining in an enterprise scenario

You can apply process mining to your enterprise using data analytics tools from Microsoft Azure. In stage 1 of the process, capture enterprise business process data and transform it into digital footprints using AppInsights. Then store the digital footprint data in Azure Data Lake for each node of the business process, including date/time stamp and event type. In stage 2, use process mining techniques to synthesize the data for insights analysis or real-time process performance monitoring. For stage 3, use the existing insights and data extracted by process mining to create intelligent tools, and enable informed business decisions with Microsoft Azure Machine Learning Studio, Microsoft Cognitive Services, and the Azure Bot Service.

How to further integrate process mining with Azure technologies

Real-time availability of digital footprints is the most important element needed for process mining. Most traditional, on-premises applications don’t have methods to obtain logs for each node of a business process. Retrieving log data on a real-time basis to monitor business process performance is difficult, but the latest version of Application Insights has the capability to perform real-time streaming of digital data directly from the source.

Application Insights

Application Insights has an API for custom events and metrics that is configurable simply by inserting a few lines of code in a legacy application. When this code is in place, we can get the digital logs of activity with date and time stamp for all activity happening in our business processes. We store these logs in Azure Data Lake, which removes the complexities of ingesting and storing logs while making it faster to get up and running with batch, streaming, and interactive analytics. Azure Data Lake works with existing IT data investments for simplified data management and governance. It integrates seamlessly with operational stores and data warehouses, so we can extend current data applications. You can also consider Azure Data Lake for multi-tenant application architecture ecosystems to centrally manage enterprise digital logs for various transactional applications.

Power BI

We can view the process mining analysis output in multiple ways depending on the user experience we’re trying to achieve. We can render the output in Microsoft Power BI for static visualization of business process flow with real-time refresh. We also use the Visio API for Power BI to create interactive process diagrams with custom handlers for click or tap events within the drawing. We can overlay the diagram with process data to track and monitor process performance in real time.

Addressing challenges in process mining

Throughout the Global Trade process mining journey, we’ve encountered challenges to applying the process mining solution. Some of the challenges have been applicable only to Global Trade, but many will be encountered by any organization that implements process mining.

• Data quality is critical. Our results are only as good as our data. We found that we had to examine our data quality up front to ensure that we were getting accurate results from process mining. We established several practices—including regular data consistency checks, creating policies for enabling appropriate logs, and ensuring that data sources were connected appropriately—to allow for relational data flow across the entire process mining dataset.
• Look for and address whitespace. We found several places in our processes where we had whitespace in the form of no data or not enough data being recorded by our process systems. For example, if we didn’t receive any information from a shipping company between when a package was picked up and when it was dropped off, that was whitespace in the continuum of our data stream. We identified whitespace in our processes and suggested fixes for the issue wherever possible.
• Don’t settle for hard-coded business rules. We found that some delays in processes were due to hard-coded business rules that were either no longer applicable, had become irrelevant, or used outdated values. For example, if our shipping department had 12 hours to pack and ship a product from the time it was ordered, they typically operated to meet that target but not necessarily exceed it. We’ve started creating and using dynamic rulesets that account for what’s happening within our processes and continuously adjust targets and metrics within the process.

Benefits

Throughout our journey with process mining, we’ve realized several benefits—specifically within the context of Global Trade, and for other teams as well:

• Reduced process time. This affects many aspects of our processes, and was arguably our biggest improvement. Once we were able to identify the issues and solve them, we found that our business was simply more efficient. We reduced cycle and lead times across our processes and reduced the overall time it takes our products to arrive to our customers.
• Increased transparency. Process mining opened up our processes to all parts of our organization. Anyone who wanted to see what our end-to-end process looked like could pull it up on a dashboard and examine the results for any products that were in the process.
• Immediate process interaction at all levels. If one of our executives sees that a high-level process is reporting below average performance, that executive can drill down into the process and identify the specific orders or processes that are causing poor performance.

Conclusion

Process mining has created a new level of insight into and control over our Global Trade business processes at Microsoft. We’re finding new and innovative ways to visualize and monitor our workflows and pinpoint room for improvement. We’re beginning to use process mining to create self-healing business processes that can provide performance feedback to the business process itself, and utilize machine learning to adjust parameters to improve process performance. By embracing and using process mining, we have the ability to dig deeply into our big data and extract meaningful, transformative insights that we can use to accelerate our digital transformation and make our business better.