How big is big data – and what can we do with it?

?Big data presents significant challenges to governments and organizations around the globe, and the variety and velocity of this data cannot be ignored. Gartner estimates that worldwide information volume is growing at least 59 percent annually, with 70-85 percent of that data as unstructured data (raw information that does not fit into an existing data model, like an SQL database). IDC predicts that cumulative data will grow 44 times over the next decade.

New processes are capturing vast amounts of both structured and unstructured data, including unprecedented amounts of new information now collected through social media sites like Twitter, Facebook, and LinkedIn. Organizations must find a way to combine internal and external data and services to effectively and efficiently collect, analyze, and make better decisions based on this data in nearly real time.

In May, Microsoft published a new brief on its big data solutions that can help national security organizations produce actionable insights from structured and unstructured data. In this post, I’d like to call out a few key sections of this paper that are particularly relevant to those working in the national security and public safety arenas.

The challenges

  • Making sense of the data explosion. Organizations need the right tools to make sense of the overwhelming amount of data generated by declining hardware costs and complex data sources.
  • Understanding a wider variety of data. Organizations need to analyze both relational and non-relational data, particularly since more than 85 percent of data captured is unstructured.
  • Enabling real-time analysis of data. New data sources, such as social media sites, are producing unprecedented volumes of data in real time, which cannot be analyzed effectively with simple batch processing.
  • Achieving simplified deployment and management. Organizations need a more simple and streamlined deployment and setup experience, ideally with fewer installation files that package the required data management tools.

The global impact

An effective solution for these challenges needs to address several aspects of the big data issue. It needs a multilayered data management approach that supports data in any form. It needs to leverage a suite of business intelligence tools that can meaningfully use analytics. Most importantly, it needs to provide deep insights that combine an organization’s data with data and services from external sources.

Earlier this year, the World Economic Forum (WEF) identified big data analysis as an incredibly useful public safety and national security tool for its ability to “connect the dots.” Building on this theme, the WEF recently released a new report, Big Data, Big Impact: New Possibilities for International Development, which explores the intersection of big data with another hot global tech trend – mobile communications – and how leveraging both technologies can improve the lives of billions of people living in poverty around the world.

Big data deals with massive datasets, but those datasets usually rely on many individual sources. This is especially the case in developing countries, where mobile phones are often the only Internet-enabled devices most people have. Mobile phone use is exploding worldwide – Cisco estimates that between 2011 and 2016, mobile data traffic will grow an average of 79 percent in Latin America, 84 percent in the Asia-Pacific region and a whopping 104 percent a year in the Middle East and Africa. 

A particularly useful aspect of mobile phone data is that it doesn’t just help create an aggregate picture of local or regional needs – it also reflects the needs and behaviors of individual phone users. This enables individuals, governments, companies, and non-profit organizations to acquire new understanding of low-income populations and develop user-centric solutions to help them more effectively. Governments, companies and non-profits could then perform data mining and analysis on the data, yielding benefits such as:

  • Faster outbreak tracking and response
  • Improved understanding of behavior changes during crises
  • Accurate mapping of service needs
  • Ability to predict supply and demand changes

The Microsoft approach

Big data analysis can support public safety in a myriad of other ways as well, from tracking botnet attacks in computers, to monitoring crime data trends, to tapping into social media and open source intelligence to help law enforcement monitor threatening online chatter. The question becomes not only to store this data, but how to also access it and analyze it for useful information on trends and patterns.

Technologies such as Apache Hadoop are able to store and analyze petabytes of unstructured data inexpensively – however, most vendors that provide Hadoop deployments operate in a silo and are not yet enterprise-ready. Microsoft offers Hadoop both as a cloud-based service on the Microsoft Windows Azure platform and as an on-premises distribution on Windows Server, enabling customers to gain business insights from structured and unstructured data of virtually any size and activate new types of data regardless of its location. These rich insights can then be combined seamlessly with the Microsoft Business Intelligence (BI) platform to provide customers with the ability to enrich their models with publicly available data and services using tools like Microsoft Office and SharePoint.

The results

With a big data solution in place, organizations realize several key benefits, including:

  • Immersive insight, wherever you are, on any data with familiar Microsoft Office and BI tools. Microsoft’s big data solution offers customers actionable insights from all of their structured and unstructured data using familiar BI tools – analyzing data with familiar tools and driving insights through simplified programming that is integrated with new JavaScript libraries.
  • Unlocking hidden value through a combination of internal and publicly available data and services, including social media sites. Microsoft’s big data solution can enable breakthrough discoveries by combining data and models with publicly available data and services, including social media sites like Twitter, Facebook, and LinkedIn. This enables customers to uncover hidden patterns using the applications and mining algorithms on Windows Azure Marketplace.
  • Support for any data, any size, anywhere through a modern data management platform that leverages the simplicity of Windows and SQL Server along with the elastic scalability of the cloud. Microsoft enables customers to seamlessly store and process data of all types, including structured, unstructured, and real-time data, through a modern data management platform. It provides the simplicity of Windows and SQL Server on Hadoop, extends data warehouses with Hadoop, and offers the elastic scalability of the cloud to big data.

In the coming months, Microsoft will be unveiling new ways in which big data solutions are supporting improved public safety and national security around the world. One recent example is Thailand’s Department of Special Investigation, which has implemented a big data solution based on Microsoft SQL Server 2012 and Hadoop software. The new solution, which manages huge data sets including more than 1 million records in both structured and unstructured formats, showed astounding improvements in efficiency – during the testing phase, data gathering and analysis that would have taken two years using the traditional approach generated results for law enforcement officers in just 15 days.

We look forward to sharing information about real-life governments and organizations dealing with the challenge of a big data explosion, and how these technologies delivered measurable results in improved citizen welfare. Please watch this space for updates!

About the Author

Mahesh Punyamurthula | Worldwide Industry Technology Strategist

Mahesh Punyamurthula has 17+ years of evangelism, architecture, and software development experience with Microsoft platforms and technologies. He has also managed technical relationships with several global safety and defense partners. Read More