Over the past few months, we’ve been writing a lot about big data in government and trends to look for down the road. For example, here is a recent post from our government colleague Parul Bhandari on why now is the best time for governments to start investing in big data. You can also check out an interesting post from regular contributor Alan Merrihew, who sheds light on the rise of sensor and machine-driven data, and the tidal wave of information that will be created by these technologies. And, finally, I highly recommend a blog post from my colleague Mahesh Punyamurthula, who recently contributed a post to the Microsoft on Safety and Defense Blog, looking at the issue of, “How big is big data—and what can we do with it?” All great reads.
For today’s post, however, I wanted to go beyond talking trends and share what I believe is a great, practical example for governments as they look to leverage big data to better execute on their missions. Recently on our website, we published a case study on Thailand’s Department of Special Investigations (DSI), and its use of big data to dramatically accelerate and improve the accuracy of its investigations into criminal cases.
Established by Thailand’s Ministry of Justice in 2002, the DSI was formed to investigate serious and sophisticated criminal activity, including transnational crimes such as human trafficking. Core to the DSI’s operations are data collection and the use of business intelligence (BI) technology to analyze criminal data for insights and to support law enforcement efforts. As part of these functions, the DSI collects massive volumes of data—both in structured and unstructured formats such as images, videos, and documents.
Before, analyzing these large and diverse data sets was proving to be a challenge for the DSI. As Deputy Director of the DSI Yannaphon Youngyuen put it, “It was very difficult to mine through the data, and the results were too broad with unclear targets.” This often forced the DSI to send personnel to actual crime scenes for investigations, which cost the agency a lot of time and money. Furthermore, mining big data was a complex process for the DSI, and often required officers to create specialized queries with the help of IT to ask specific questions of data to uncover important tips related to criminal cases.
In response to these challenges, the DSI sought out better BI and data-mining tools to reduce complex and manual processes and establish a system that could automatically notify personnel of suspicious persons or activities related to criminal cases. For example, when many foreign criminals pour into the country and all travel to the same location, or when there is a noticeably large sum of money being transferred into the country.
At the core of DSI’s new big data solution are Microsoft SQL Server 2012, which DSI uses for importing and analyzing its huge volumes of structured and unstructured data from multiple sources, and Apache Hadoop software, an open-source platform that DSI uses to store large volumes of unstructured data. With the new system in place, the DSI has seen a variety of benefits, including:
Enhanced BI capabilities with Microsoft SQL Server 2012 Power View, an interactive data visualization and exploration tool that the DSI now uses to quickly create and present reports to high-ranking officers that include analysis details and an executive summary of each case being monitored
Automatic notifications of suspicious incidents—based on predefined sets of conditions established by DSI personnel—to support criminal casework
Eliminated reliance on IT to produce reports and conduct queries of data related to investigations
However, what’s most impressive about DSI’s new big data implementation is the dramatic impact that it’s had on investigations overall. Before, conducting a traditional investigation could take as many as two years to search for tips, gather, and analyze data. Today, with the Microsoft big data solution, the DSI is able to conduct investigations in as few as 15 days, thanks to the automated processes, reduced complexity and more accurate insights afforded by the new system. If you’d like to learn more about this impressive case study, be sure to visit our website.
Have a comment or opinion on this post? Let me know @Microsoft_Gov. Have a question for the author? Please e-mail us at firstname.lastname@example.org.