Azure Data Explorer analyzes petabytes of data to power Microsoft 365 Defender’s integrated threat security

Thousands of organizations use Microsoft 365 Defender daily to keep their digital estate secure. The enterprise defense suite safeguards endpoints, applications, emails, user identities and data. It prevents, detects, and remediates cyber threats, and enables organizations to build a complete picture of attacks in a single unified portal. “For extended, multi-phase attacks such as ransomware, Microsoft 365 Defender must proactively search for the evidence of an attack in progress and take action to stop it before it is completed,” says Yael Flashner, Principal Group Engineering Manager at Microsoft. To enable this level of proactive protection, her team uses Azure Data Explorer, a fully managed data analytics service for real-time analysis of huge volumes of streaming data.

A better way to balance user experience and security

“When we first built the Microsoft 365 Defender platform, Elasticsearch was a natural choice as it was a trusted technology,” says Flashner. However, after trying the Elasticsearch platform, the team still encountered challenges with cluster management and the introduction of new features. “Because this is a customer-facing product, we needed to make sure our users have a very responsive experience, regardless of the number of events occurring in their networks,” explains Flashner. To do this, the team sought an alternative that would first complement and ultimately replace Elasticsearch. “We became one of the first adopters of Azure Data Explorer (ADX) even before it was an official public offering,” says Flashner. 

Integrated protection from sophisticated threats

ADX uses an easily readable and understandable Kusto Query Language (KQL) to provide interactive analytics capabilities over diverse structured and unstructured data sets. “KQL is very intuitive and expressive, enabling users to parse and apply different transformations on raw and unstructured data to speed up development,” says Flashner. In addition, the simplicity of the language was reflected in a minimal learning curve, allowing Microsoft 365 Defender developers to embrace the platform’s benefits from the start. 

The flexibility of ADX to support simple and complex queries, as well as the high performance of the language, unlocked opportunities to evolve the platform. The platform brings together our security products for endpoints, Office 365, Cloud Apps, Identities and more that were all initially built with Elasticsearch—as individual services. “Instead of multiple products, we wanted to provide a unified solution that protects enterprise customers from cyberattacks from different attack vectors,” says Flashner.

To integrate all security products into a single unified platform and make clusters easier and less costly to manage, Flashner and her colleagues redesigned the underlying interactive analytics data platform to leverage ADX. The new technology has resolved issues with sudden spikes in performance and improved the data ingestion rate by approximately 12 times.

Proactive search for cyber threats

Fast and scalable, ADX has enabled the creation of a query-based threat hunting  experience for security teams to trace suspicious signals and behaviors and leverage their unique organizational knowledge. “We built a dedicated page called Advanced hunting that lets you explore 30 days of alerts and raw data. Customer SOC teams use it to write custom KQL queries and search for signs of attacks across their organizations,” says Flashner. 

The Defender analytics platform uses cross-cluster KQL queries enabling end users to proactively and independently inspect events across their network, applications, endpoints, and accounts to prevent sophisticated threats. “Flexible access to the Microsoft 365 Defender data enables unconstrained hunting for known and potential threats,” adds Flashner. “Customers appreciate this feature as a critical piece of the portal.” 

One platform to aggregate all security data

Business users can also leverage KQL to extend defenses with Microsoft Security tools. Having the same formatting and query language across the different tools is making data hunts more efficient, focused, and effective. “In the past, we were jumping into multiple tools, trying to paste timelines together,” says Joe Lykowski, Cyber Defense Leader at Dow Inc. His organization is one of the many enterprises that use Microsoft Defender.

Previously reliant on other query languages, Lykowski’s team now uses KQL in hunting, alerting, SOAR (security orchestration, automation, and response) activities, and custom programs that go against the graph. “Using the same formatting for different tools means analysts save time from having to translate similar query languages from tool to tool,” he says. “It was initially challenging to move to KQL, but our analysts now prefer it over other languages.”

Another advantage of Microsoft 365 Defender is that the platform combines data aggregation with real-time events processing - ingesting petabytes of data at a time. Equipped with a correlation solution, it processes trillions of signals daily, providing a comprehensive view of attacks spanning different alerts, suspicious events and impacted assets. “The Microsoft 365 Defender portal correlates related alerts across all products into a single incident. This makes it much more effective and enables users to take actions quickly,” describes Flashner.  

More room and time for product innovation

Replacing the prior technology with ADX provided the Microsoft team with numerous benefits. “The development speed increased as we started leveraging Kusto, which directly impacts time to market,” says Flashner. In addition, because the entire solution is available as a PaaS service in Azure, it doesn’t require a dedicated team to support it. With no deployment overheads and significantly reduced support, the running costs of maintaining the data exploration infrastructure decreased by more than 50 percent.

But the economies of platform-as-a-service aren’t the only savings. “Kusto compresses data extremely well, reducing overall size of clusters that lowers the service's total cost,” adds Flashner.

Free from the hassle of managing the platform’s infrastructure, developers can turn their focus to other critical tasks. “With Elastic, it was sometimes challenging to meet the defined SLAs for Microsoft 365 Defender. We had to allocate our own resources to handle live-site issues,” explains Flashner. “In contrast, being a fully-managed service means that ADX doesn’t require dedicated DevOps for support and comes with 99.9 percent availability, making it a more effective choice.”

The language and the platform keep evolving and innovating in line with the users’ needs. With ADX, the team now has better visibility of threats to their networks and works much more efficiently, yet, there are more functionalities to explore. “We are currently rolling out materialized views, which aggregate big data entities for efficient and convenient access,” explains Flashner. “We are also experimenting with preview features and adding graph query capabilities.” Moving forward, her team plans to expand the correlation platform capabilities, supporting more data and new workloads and domains to keep company data and assets secure with integrated threat protection.