Taming big data to improve national security

Whether it’s connecting the dots in online chatter to ferret out extremists or bolstering preparedness and response to large-scale disasters – analyzing increasingly massive, complex data sets to discover actionable intelligence and insight has become integral to 21st century public safety and national security.

The World Economic Forum this year identified big data analysis as an incredibly useful tool for public safety and national security for its ability to connect the dots. National security organizations must be able to share real-time situational awareness to succeed at their missions. Even one additional nugget of actionable information—delivered to the right people at the right time in the right way—can mean lives saved instead of lost.

Challenges abound, though, in incorporating big data into existing public safety and national security operations. The amount of data that organizations can access and use has increased exponentially in recent years, thanks in part to the global proliferation of mobile, Internet-enabled devices. These organizations must create a common operating picture that incorporates targeted, real-time insight from complex data from multiple national and international stakeholders. Avoiding data overload and staying ahead of evolving, escalating threats require innovation and vigilance.

To help our public safety and law enforcement clients protect themselves and their communities better, Microsoft has incorporated big data capability and compatibility into many of its national security solutions. These technologies are helping decision-makers at all levels—from first responders to national and international policy makers—more easily discover, understand, and implement real-time actionable intelligence.

For instance, Microsoft’s Big Data Solution integrates with Microsoft Business Intelligence (BI) tools as well as familiar productivity applications such as PowerPivot and Excel. It’s available via Windows Server 2008 R2 and through the cloud via Windows Azure. These technologies directly enable predictive analytics, advanced detection systems, and other capabilities that prioritize resources and manage risk.

Big data plays into other key technologies that augment public safety and national security. Data visualization features in Microsoft SQL Server 2012 and Microsoft Visio 2010 enable layering of multiple data types onto geospatial maps and other intuitive interfaces so people can connect the dots among seemingly unrelated data points. Cloud computing is helping public safety, law enforcement, and related agencies handle even larger amounts of data while enabling more powerful collaboration capabilities. The list of relevant technologies goes on: video, mobile, mapping, natural language processing, pattern recognition, and machine learning, to name a few.

Social media is a growing area of interest for national security, as the technology’s central role in the 2011 Arab Spring showed. Microsoft is helping social media companies develop technologies that could have strong impact on national security.

An excellent example is Klout, an online service that tracks the social media influence of more than 100 million clients, to enable them to better understand and leverage that influence. Klout’s VP of Engineering, David Mariani, wrote on Klout’s blog about how the company needed a custom solution to perform real-time BI queries on 35 billion rows of data—definitely big data territory—with an average response time of less than 10 seconds.

Another innovative tool currently in development is Microsoft Codename Data Explorer. This informative video explains how this cloud-based collaboration and data aggregation tool enables users to gather, analyze and visualize data. What’s more, it recommends other data that might interest users, such as crime statistics. To learn more, you can sign up to try the Microsoft Codename Data Explorer Cloud service or, if you want to get really technical, read this paper.

Have a comment or opinion on this post or a question for the author? Let me know @MicrosoftPSNS or email us at safetyanddefense@microsoft.com.