Editor’s note: We’ve republished this blog with a new companion video.

Sometimes you outgrow the capabilities of a well-loved tool—that’s exactly what happened to Microsoft and its on-premises Security Information Event Management (SIEM) system. Thanks to a timely assist from Microsoft Sentinel, the company hasn’t missed a beat.

Our old SIEM capped out at 10 billion events daily. We had already begun to leverage other solutions to keep increasing our security monitoring coverage.

– Mei Lau, principal PM manager, Microsoft Security

As an enterprise, Microsoft’s footprint is massive. The company sees a lot of malicious traffic, which results in more than 20 billion cybersecurity events per day. This massive wave of noise was hard to sort through to find real threats—until the company’s internal security team turned to Microsoft Sentinel, which, thanks to the cloud and AI, has the power to keep up with that volume.

“Our old SIEM capped out at 10 billion events daily,” says Mei Lau, principal PM manager for Microsoft Security, the organization that powers, protects, and transforms Microsoft. Lau is responsible for leading the migration of Microsoft’s legacy SIEM to the cloud-based Microsoft Sentinel. “We had already begun to leverage other solutions to keep increasing our security monitoring coverage.”

Because running out of capacity could lead to a worst-case scenario, Lau’s team works with the Microsoft Sentinel product group to test and pilot the new security monitoring system, which includes several time-saving and modern solutions that empower security analysts to connect to and query datasets quickly and easily.

“Ingesting data into our legacy SIEM took hours,” Lau says. “In Microsoft Sentinel, it takes around 10 minutes, which is 18 times faster.”

Now, they have deployed the cloud-based version of SIEM throughout Microsoft’s internal Security Operation Centers (SOC). In partnering with Microsoft Security, which provides enterprise IT capabilities across Microsoft (including security), the Microsoft Sentinel team introduced several time-saving and modern solutions that empower security analysts to connect and query datasets quickly and easily. Best of all, they’re using the power of cloud computing at scale.

[Discover how Microsoft protects its network with Zero Trust. Find out how Microsoft uses elevated-privilege accounts for security.]

For a transcript, please view the video on YouTube: https://www.youtube.com/watch?v=dtyDMjMvN98, select the “More actions” button (three dots icon) below the video, and then select “Show transcript.”

Getting it right with the right partners

The Microsoft Sentinel product team tapped the expertise of the company’s internal security team in Microsoft Security for insights about how to improve the product. Their input helped shape Microsoft Sentinel into a SIEM that dramatically improved how efficiently it responds to threats.

If we can help them be successful, we’re also helping our large customers, who often have the same challenges, requirements, and needs.

– Laura Machado de Wright, principal PM manager, Microsoft Sentinel product team

“Microsoft Sentinel uses all the automation and scalability capabilities available in the Azure platform,” Lau says.

Microsoft Security’s engagement with the Microsoft Sentinel team addressed two sets of needs at once.

“They get the benefits of Microsoft Sentinel for incident response, but we get the benefit as the product team of working with customers, like our own internal digital security team,” says Laura Machado de Wright, a principal PM manager on the Microsoft Sentinel product team. “If we can help them be successful, we’re also helping our large customers, who often have the same challenges, requirements, and needs.”

The collaboration meant the product team could identify what enterprise-scale customers were looking for at a faster rate.

“We can work closely and iterate more rapidly with internal teams,” Machado de Wright says. “We can get their requirements and feedback before moving into formal previews with external customers.”

These early interactions allowed the product team to work through a few nuances that could have disrupted users. In an early version of Microsoft Sentinel, for example, some of Microsoft Security’s security analysts noticed that they were getting a lot of long notifications.

“When you start testing, you realize you need certain capabilities,” Lau says. “We were able to point out the business impact of noisy alerts that are too long.”

In response, the product team introduced suppression and aggregation support to avoid alert fatigue, reducing the amount of noise generated by Microsoft Sentinel.

“Now we have a better product that meets our needs at an enterprise level,” Lau says.

Always a group effort

One objective of Microsoft Security is to unify security operations teams onto a single SIEM—Microsoft Sentinel. “Depending on the scope, there are different teams responsible for protecting Microsoft,” Machado de Wright says. “There are some common solutions between them, but many security operations teams built their own solutions or relied on third-party solutions to manage security events. With Microsoft Sentinel, we think there’s an opportunity for them to be the first and best customers of Microsoft.”

With Microsoft Sentinel, it’s easier for SOCs to develop a tactical and coordinated response to security threats and incidents.

“Even though they might look at different pieces of the puzzle, data from different internal teams can be brought into Microsoft Sentinel and create detections,” Machado de Wright says. “Then, automation can assign it to the right group.”

These multiple sources can be connected for rich, multifactor detections.

“Multifactor allows us to grab from multiple sources and compare them together,” Lau says. “We can see if someone is attacking us in several different ways. Between detection and hunt, it’s very simple to track down what’s happening.”

Unifying security operations teams onto the Microsoft Sentinel platform also allowed the company’s internal security team in Microsoft Security to align on a deployment strategy.

“It was great to work with other SOCs within Microsoft,” Lau says. “We have the shared goal of protecting the entire enterprise, which enabled us to identify key requirements for parity to retire the legacy SIEM.”

Steps had already been taken to retire the legacy SIEM, so deploying Microsoft Sentinel in a timely manner was critical.

To move to Microsoft Sentinel, the product team needed to verify that equivalent features and capabilities were live in the new security environment. Making sure the various teams’ needs were aligned helped ensure that.

“Some of these teams had fairly mature monitoring systems,” Machado de Wright says. “We had to work on prioritization and work closely to understand their scenarios to meet the requirements of their timeline.”

Faster, together

To build new detection systems, you need connected data sources. But first, you have to find each source and connect it to your analytics engine.

“Before, you had to understand how the data was structured and then build software to connect to your events management system,” Lau says. “Microsoft Sentinel’s broad ecosystem allows many out-of-the-box data connectors to be connected up to 18 times faster.”

This is one of the major ways Microsoft Sentinel accelerates and empowers engineers and analysts.

“Finding access to data can be ponderous across large volumes of data,” Lau says. “When security analysts go in and perform open-ended queries to find access to data in the repository, Microsoft Sentinel is extremely fast.”

Now tracking down a new connector or data source in Microsoft Sentinel takes just a few seconds. This free time has allowed the security team in Microsoft Security to reprioritize engineering resources previously dedicated to scaling the infrastructure. Plus, the time-saving automations introduced with Microsoft Sentinel have improved the lives of Microsoft Security’s SOC analysts.

Some of these time savings manifest in how quickly code can be written and deployed.

“It all happens at the speed of pushing code to the cloud,” Lau says. “So, a matter of minutes.”

This streamlined process gives Microsoft Security much better change control, enabling a continuous integration and continuous detection pipeline.

Transforming the future of security

Microsoft Security isn’t the only group benefiting from Microsoft Sentinel.

During development, Microsoft Security and the Microsoft Sentinel product team also solicited input from other enterprise customers. These partners, including a global retailer that experiences more than 9 billion security events per day, helped shape the final product.

“Sometimes we get conflicting feedback from customers,” Machado de Wright says. “We can’t always address it, but we can dive deeper by asking the internal team if they have the same pain point or scenario.”

Thanks to the contributions of Microsoft Security and its partners, the Microsoft Sentinel team has quickly developed and released a product that can handle the scale and security needs of modern enterprises.

“We have access to different personas, like analysts, engineers, managers, and different security operations teams,” Machado de Wright says. “The ability to just sit with them accelerated everything.”

And there’s still more to discover with Microsoft Sentinel.

For example, with new ways to engage and interact with connected datasets, Microsoft Security is now using machine learning with the new tool. “We are moving some of our most complex detections into Microsoft Sentinel,” Lau says.

For enterprise customers like Microsoft who already have the Microsoft Azure stack, using cloud-based security tools made a lot of sense.

“We’re already using Azure,” Lau says. “Now we have a better product that meets our security needs at an enterprise level. Our security operations teams don’t need to leave Microsoft Sentinel. They can query different Azure Data Explorer clusters and other workspaces with permission. It’s a single pane of glass to complete an investigation.”

• Discover how Microsoft protects its network with Zero Trust.
• Find out how Microsoft uses elevated-privilege accounts for security.

Tags: Microsoft Azure, security