Data Deluge and Digital Watersheds
Commodity sensors and Internet connectivity have created a veritable data deluge. Yet it remains a challenge to find, access, clean, and reuse data. That is particularly true when data from different sources is needed for synthesis science—bringing many diverse observations together to create a larger, holistic view.
Scientists at the Berkeley Water Center, led by James Hunt, worked with the National Marine Fisheries Service on a synthesis challenge in the Russian River area. The watershed is the breeding ground for several species of fish, but wine-grape farming, urbanization, gravel mining, and other factors have affected the river. As a result, the fish have become endangered, and habitat restoration is critical.
To enable such studies, Microsoft Research’s Catharine van Ingen and Lawrence Berkeley National Laboratory researcher Deb Agarwal built a digital watershed. Constructed on a Microsoft SQL Server database and SQL Server Analysis Services data cube, the digital watershed enables simple interactive browsing of the assembled diverse sensor and field observations. Data updates are automatically harvested from available government websites and are ingested from smaller spreadsheets or other sources. Some of the data are historic, dating back more than 100 years. Others are real-time measurements with only transient availability over the Internet.
Among the questions examined with the digital watershed was the impact of “frost dips.” During the season when wine-grape buds are setting, local farmers use sprinklers to avoid frost damage. Pumping from the river to supply the sprinklers causes transient dips in the river water level. Such dips can strand small hatchling fish, making them susceptible to predators or oxygen deprivation.
This research led to a fundamental change in how hydrologists get timely answers to their questions by using advanced computing methods. It established a model for how scientists can use technologies to help them solve problems with data processing so that they can remain focused on their sciences.
Learn more about this research: