The National Climatic Data Center (NCDC), a division of the National Oceanic and Atmospheric Administration (NOAA), manages the world’s largest archive of climate and weather information. NCDC provides support for the study of global climate change and other environmental issues. One of the center’s major initiatives is to digitize its vast archive of paper records and make them available to researchers over the Internet. NCDC previously used a records management system developed, managed, and hosted by Microsoft® Gold Certified Partner National Interest Security Company (NISC). While pleased with the NISC system, NCDC wanted a solution it could manage and host locally, if necessary. Working with NISC, NCDC used Microsoft Office SharePoint® Server 2007 to develop a simple, reliable records management system that can handle vast amounts of data and that its users can adopt seamlessly.
||The data we’re managing with Office SharePoint Server 2007 will help us make more accurate weather and climate forecasts. That will reverberate throughout the economy.
CDMP Program Manager, National Climatic Data Center
The National Climatic Data Center (NCDC) is a division of the National Oceanic and Atmospheric Administration (NOAA), an agency of the United States Department of Commerce. NCDC, headquartered in Asheville, North Carolina, manages the world’s largest archive of weather data. The center stores 99 percent of NOAA’s known data, including more than 320 million paper records and 1.2 petabytes of digital data.
The more than 200 years of weather and climate data stored at NCDC provide an historical perspective that is vital to the study of global climate change and other important environmental issues. The work of NCDC supports transportation, shipping, agriculture, and other major elements of the U.S. and world economies.
A major initiative of NCDC is the Climate Database Modernization Program (CDMP), a project to digitally scan and index archived paper and microfilm records, and make them available on the Internet to climate and environmental researchers. Since 2000, NCDC has been working with National Interest Security Company (NISC) —formerly Information Manufacturing Corporation—to scan, index, and store more than 54 million documents—nearly 12 terabytes of data—in a proprietary records management system developed by NISC.
In addition to developing the records management system, NISC—a Microsoft® Gold Certified Partner—has been responsible for scanning and indexing the documents in the NCDC archives, uploading them into the system, and then hosting the Web-based application from its own data center. NCDC personnel review the index metadata associated with the documents and make revisions if necessary.
Using the system, researchers at NOAA, other government agencies, and academic institutions can search, view, and download documents such as surface weather observations from a network of 8,000 stations across the United States, international weather data, private journals, or ship logs. Some of the documents date back to the mid 1700s.
The system—and the partnership between NCDC and NISC—has been extremely successful in contributing to important climate research around the world. However, NOAA was concerned that if the CDMP lost part or all of its funding, NCDC would not be able to independently manage the NISC records management system. NCDC needed a commercial records management solution that it could operate and manage, but that would retain the powerful indexing and search capabilities of the NISC system.
The agency wanted a cost-effective solution that could be easily transitioned to in-house management should the need arise. NCDC obviously needed a system that could reliably handle vast amounts of data, and if it could, it wanted to find technology that would allow it to add collaboration, workflow, and other enterprise capabilities to the system. But most importantly, NCDC wanted a solution that would recreate as closely as possible the interface of the existing system. It did not want to have to retrain its own personnel to use a new system, and it needed a solution that thousands of researchers around the world could adopt seamlessly.
“We wanted a new records management solution with the least amount of pain possible,” says Tom Ross, CDMP Program Manager at the National Climatic Data Center. “We have a lot of users depending on the system on a daily, even hourly basis. They’ve been used to a particular interface for 10 years, and we needed a new system that would be just as familiar.”
In 2006, NCDC began evaluating records management solutions, including products by Meridio, Documentum, and Microsoft. Both NCDC and NISC were impressed by the capabilities in Microsoft Office SharePoint® Portal Server 2003, but the solution had to be certified to comply with a federal government DOD 5015.2 standard for document management, and Office SharePoint Portal Server 2003 was not 5015.2 certified.
||We wanted a new records management solution with the least amount of pain possible. We have a lot of users depending on the system on a daily, even hourly basis.… and we needed a new system that would be just as familiar.
CDMP Program Manager, National Climatic Data Center
However, during the evaluation period, Microsoft Office SharePoint Server 2007 was 5015.2 certified. Combining the certification with the readily available support and ubiquitous familiarity of the Microsoft solution, and the cost-effectiveness of the Microsoft Service Provider Licensing Model, NCDC quickly decided to move forward using Office SharePoint Server 2007.
Also, because Office SharePoint Server 2007 is built on the Microsoft .NET Framework—which NISC used to develop its proprietary system—NISC could easily customize Office SharePoint Server 2007 to maintain the look and feel of the existing records management interface. Using Office SharePoint Server 2007, NISC developed a new records management system for NCDC called the Environmental Document Access and Display System (EDADS). “By using SharePoint Server 2007, we were able to build upon our existing development experience with the Microsoft .NET Framework,” says Eric Wagstaff, Lead Software Engineer at NISC. “This allowed us to deliver the user interface required by NCDC in a timely manner.”
With Office SharePoint Server 2007, NCDC can still take advantage of NISC hosting services. At its own site NISC can manage the system, execute back-ups, updates, and physical security for the system servers. NCDC can also take advantage of NISC’s investments in storage capacity for the more than 12 terabytes of data in the system. NCDC stores millions of scanned and indexed documents in EDADS, organized in about 125 SharePoint libraries.
Each library is associated with its own set of index metadata, such as the station name, date of the observation, where the data was collected, and the country, state or region where the observations were made. Meteorological experts at NCDC review and develop index themes for paper and microfilm records in the NCDC archives. The documents are then passed to NISC personnel for scanning, indexing, and uploading into specific libraries.
End users can access the information in EDADS in two ways. At the NCDC Web site, the general public can access some of the libraries through the Image and Publications System developed in-house by NCDC and linked to the SharePoint libraries in EDADS through Web services written by NISC. The users can find information, including NCDC climate publications, by searching drop-down menus. Then, by clicking a document name, they can view and download a Portable Document Format (PDF) file of the record.
U.S. government employees and their contractors, academic institutions doing environmental research, and other researchers associated with NOAA projects can register for accounts and work directly in EDADS. To meet NCDC’s specific business requirements, NISC customized the search functionality in Office SharePoint Server 2007 to optimize user queries. “There are 54 million records in the system so far,” says Cindy Karl, IT Specialist at the National Climatic Data Center. “We have a lot of users out in the world who need to find specific documents, and we have to make sure they can find them.”
Researchers can choose a collection, which opens a list of document libraries. When the user selects a document library, he or she enters a search page, which displays a number of index fields based on the metadata associated with the library; for example, the document ID number, date, or location.
|Figure 1 - With NCDC’s Environmental Document Access and Display System,|
users can search and then display millions of weather and climate documents,
along with any associated index data and tools to view, save, or print the documents.
The user can then select one or more index fields and enter their search criteria. If a user does not have the specific search criteria, he or she can choose a “wildcard” search. For instance, if the user wants to retrieve an entire month’s worth of data, he or she can enter the month/year portion of a date (for example, “01/1982”) in the Date index field and choose Partial Date Wildcard. The search will return results for all days in the month of January, 1982. Entering just the year would result in the return of an entire year of documents. In the Advanced Search, users can narrow or broaden their search by identifying parameters for each criteria, such as less than, greater than, equal to, or between; or by connecting selected index fields with “and” or “or.”
The search returns a list of results from the specific document library selected. By clicking the View command, the user can see a View pane that displays the document and the index data for that document, along with tools to view, save, or print the document; return to the results page; or view the previous or next result in the list (Figure 1). Some internal NCDC users, based on their account settings, can edit the index data associated with the document.
When NCDC used Office SharePoint Server 2007 to develop EDADS, it created a simple, reliable records management system that it could manage in-house and that both internal and external users can adopt with a minimum of training and organizational pain. Its new system has the capacity to accommodate the large amounts of data in the NCDC archives and will be able to scale up to accommodate even more records.
In addition, NCDC will be able to use Office SharePoint Server 2007 in the future to add collaboration, workflow, and other enterprise capabilities to the system. Most importantly, with its new records management system, NCDC will continue supporting important weather and climate research throughout the world.
Hosted Self-Manageable Solution
By developing a new records management system based on Office SharePoint Server 2007, NCDC now has a solution that it can manage and operate itself. NCDC plans to continue its productive relationship with NISC, but with the new solution, NCDC personnel could take over the tasks now being performed by NISC if necessary.
Because NISC was able to easily customize Office SharePoint Server 2007 to closely replicate the user interface of the legacy records management system at NCDC, researchers throughout NOAA, other agencies, and at institutions around the world can adapt to the new system easily and quickly.
||There are 54 million records in the system so far. We have a lot of users out in the world who need to find specific documents, and we have to make sure they can find them.
National Climactic Data Center
Because Office SharePoint Server 2007 interoperates with the integrated publication system on the NCDC Web site, the general public can access NOAA weather and climate information in the same way they have in the past.
Large Data Capacity
So far, NCDC has made available more than 54 million documents in EDADS—almost 12 terabytes of data—and that is just the beginning. The NCDC archives include approximately 150,000 boxes of paper documents. Only about half have been digitized so far. In addition, the CDMP will likely digitize and upload documents from additional NOAA bureaus and other sources, including other U.S. agencies and foreign governments.
“I get different proposals every year from people within NOAA to rescue and archive data from systems in the fisheries, oceans, and other services,” says Ross. “Developing a system with Office SharePoint Server 2007 that could potentially handle a doubling or tripling of the records we may come up with over the next 10 years has been very important.”
Support for Important Research
NCDC is the nation’s official archive of weather and climate data. By integrating historical data into the climate record, NCDC provides researchers better data sets with a longer time series, so they can make better conclusions. In just one important example, the data available on EDADS will help efforts to track, understand, and combat global climate change.
“In many cases, this is life-and-death information we’re giving people,” says Karl. “For example, hurricane, tornadoes, and other extreme weather events are affecting us more and more. By providing data from past events, we can improve hurricane forecasting, support research into predicting tornadoes, and enhance climate-change modeling.”
The consequences of these types of research activities cannot be overestimated. “It’s been estimated that weather and climate affect one-third of the U.S. gross national product,” says Ross. “The data we’re managing with Office SharePoint Server 2007 will help us make more accurate weather and climate forecasts. That will reverberate throughout the economy.”
Microsoft Office System
The Microsoft Office system is the business world’s chosen environment for information work, providing the programs, servers, and services that help you succeed by transforming information into impact.
For more information about the Microsoft Office system, go to:
For More Information
For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:
For more information about National Interest Security Company (NISC) products and services, visit the Web site at:
For more information about NOAA National Climatic Data Center, visit the Web site at: