Big Data making a ‘big entrance’ on the worldwide public sector stage

19 April 2012 | Joel Cherkis, General Manager, Worldwide Government

These days it’s hard to pick up a newspaper or trade publication without reading at least one story about Big Data. The term refers to the massive—and exponentially rising—amounts of data that organizations today are collecting in their data warehouses. Pouring in from a variety of sources (e.g., sensor data, geospatial data, digital records, web traffic, and e-commerce), the amount of data generated from these sources not only presents a storage challenge, but more importantly, a fundamental challenge of how to make sense of all this information, and mine it for value.

According to a 2011 study by International Data Corp (IDC), a market-research firm, humans produced and replicated 1.8 zettabytes of data worldwide in 2011. To put that number into context, technology blog Mashable offers an infographic comparing this volume of data to the equivalent of 200 billion HD movies—at 120 minutes each. According to this figure, it would take one person 47 million years of 24/7 viewing to analyze that amount of information. This all goes to say that analyzing Big Data is truly a massive undertaking. However, it’s an endeavor that holds tremendous value for organizations in private industry, as well as the public sector.

In the not-too-distant future, I believe that Big Data will become inextricably linked to how public sector organizations operate and carry out their missions. From using Big Data to identify patterns that can help boost efficiencies and eliminate waste, to analyzing and combining sensor datasets to enhance programs ranging from environmental quality to national security, there are countless applications for Big Data in government.

And, this data-driven mentality will only become more ingrained as advanced uses of data, such as predictive analytics, continue to evolve and mature. In fact, this is the driving force behind Hadoop for Windows Azure, Microsoft’s recently announced big data solution that allows government organizations to unlock new insights from both structured and unstructured data. Optimized for SQL Server, Microsoft’s big data service distributes enterprise-ready Hadoop solutions on Windows Server and Windows Azure.

Today, Big Data is already being applied to tackle common public sector challenges such as traffic congestion and environmental quality. Take, for example, our Microsoft Research division’s Clearflow project, which applies machine-learning techniques to huge volumes of traffic data to help sense, analyze, and forecast future traffic flows.

Originally framed to address frustrations encountered while navigating traffic around our headquarters in Seattle, Washington, the project has since led to a fielding of Clearflow to major cities across North America, where Clearflow analyzes tens of millions of road segments across North America. Based on this data, Clearflow works to answer questions such as: How long until a current traffic jam disappears? How soon until open flows on a highway system become congested? And, much more.

From the public sector perspective, by better understanding the causes—and predictability—of traffic patterns, governments stand to achieve a variety of goals, from improving air quality associated with traffic flow to making more strategic investments in transportation infrastructure (based on traffic behavior).

Another application of Big Data, which we’ve blogged about before, is the European Environment Agency’s (EEA’s) Eye on Earth application, built on our Windows Azure and Microsoft SQL Azure platforms. The application was created to make environmental data more accessible for European citizens and policy makers, with the goal of keeping them more informed and engaged. The application collects a vast amount of environmental data across the EEA’s 32 member countries (water quality indicators, air particulate matter, geospatial data, etc.) and presents this information on an interactive map using Bing Maps. The result is an informative, interactive application that serves up a large set of actionable data. As Big Data analytics continues to grow within the public sector, we’re excited to see similar outcomes—made possible by governments unlocking the full potential of their data.

Have a comment or opinion on this post? Let me know @Microsoft_Gov. Have a question for the author? Please e-mail us at

Joel Cherkis
General Manager, Worldwide Government

About the Author

Joel Cherkis | General Manager, Worldwide Government

Joel leads a team of business development and technology professionals supporting policy decisions and the delivery of relevant and scalable technology solutions into public sector markets around the world. Read more