Climavision analyzes more data to provide enhanced weather forecasting using Azure HPC + AI

Forecasting from the ground up

Climavision’s enhanced weather forecasting is rooted in its belief that accurate weather predictions demand more comprehensive datasets, gathered from more places, more rapidly. Climavision Chief Scientist Peter Childs likes to use a baseball analogy: Climavision starts playing in the first inning, while traditional weather forecasters only join in the fifth.

The company achieves its advantage by augmenting and supplementing publicly available, government-provided weather data—datasets other weather service providers are limited to—with its own comprehensive, proprietary observations and data. Climavision’s national radar network sits between existing government infrastructure and fills in the blind spots. With its own observations, the company can deliver more comprehensive weather information in real time. Climavision also uses space-based observations that provide a more thorough view of the middle to upper atmosphere, which is incredibly insightful for medium- and long-term forecasting, enabling a comprehensive global model that extends 16 days in the future. “All the information from the radar and satellites that we’ve deployed plus all the other global weather platforms amounts to roughly 3.5 billion weather observations coming into our system daily,” says Climavision Founder and Chief Executive Officer Chris Goode.

This staggering amount of data requires serious resources for collection, transmission, storage, and processing. From that 3.5 billion observations aggregated from thousands of sources and providers, Climavision quality controls, processes, and stores tens of millions of refined weather observations for its forecasting cycles. To conduct leading-edge and highly accurate forecasting, it’s also essential that Climavision quickly and efficiently extracts maximum value from its data. The company uses tightly coupled compute resources to accomplish this. “We feed data into sophisticated forecasting engines that basically solve the fluid equations of the atmosphere,” says Childs. “But to quickly solve those at impactful resolutions, we need tightly coupled workloads, which means that computers need to communicate with each other very quickly.” To do this, Climavision selected Azure HPC + AI.

Childs and his team have long-term experience using Azure, having moved high-performance, on-premises processing resources to Azure about six years ago. “Azure was the first cloud compute platform using a high-performance communication technology between computers that gave us the resources we needed,” notes Childs. “After successfully standing up our global workload application in 2016, it’s only gotten better since then.”

From in-house supercomputers to high-performance cloud computing

The Azure HPC + AI platform provides the infrastructure and orchestration that Climavision’s resource-intensive workloads need. The data that the company gathers globally is aggregated across Azure Virtual WAN and fed into Azure Data Lake Storage. Climavision’s proprietary platform, Dalton AI, provides connectivity capabilities through Azure Data Factory and Azure Event Grid, linking Climavision’s APIs and acting as an interface component for its dashboarding. Dalton AI filters and delivers accurate, up-to-the-minute weather forecasting information specifically tailored to each of its customers. “We use Azure Data Factory to carve the data into the bits we need, with Azure Event Grid generating related events to alert and trigger visualization and analysis pipelines,” says Climavision Chief Technology Officer Jon van Doore. “The whole solution end-to-end is proactive and event-based.”

Climavision’s primary environment, where it runs its data collection and quality control, its global forecasting applications, its point forecast system, and its graphical generation is underpinned by reserved instances of Azure HBv3 virtual machines using third-generation AMD EPYC™ processors. Climavision uses HBv3 instances now because their density expertly matches and supports the company’s tightly coupled workloads. “Having that density is really helpful for us from a computational and a cost perspective as well,” states Childs. Climavision is planning on expanding its cloud footprint to more fully harness the enhanced performance of its HBv3 virtual machines, which are now enabled with AMD 3D V-Cache™ technology available on third-generation EPYC processors.

In addition to its use of Azure Virtual Machines, Climavision draws on other key Azure solutions to undergird its infrastructure, including the Web Apps feature of Azure App Service to connect its radar network, Azure Functions to index and move data, and Azure Cosmos DB to index and handle product database requirements. The company also uses Azure App Service for its applications, which it will soon be updating to Azure Kubernetes Service, API Management for all of its customer facing APIs, and Azure Blob Storage for the majority of its storage requirements. ”Azure storage enables the Climavision cloud’s backbone, moving petabytes of data in and out of our forecasting systems and radar networks,” says Van Doore. “It provides us the flexibility to optimize for speed when more performance is needed or cost when data is no longer fresh, all backed by highly secure, geo-redundant storage with granular access control.”

With the help of its Azure infrastructure, Climavision can now move as quickly as the weather changes. “The biggest gain we get from Azure HPC is time to market. It’s lightning quick,” says Van Doore. “We’re much faster and much more agile because we can just hit a button, spin up a service, and start connecting. With Azure, our radar viewer was up in less than two weeks and our dashboard in less than four days—these are very fast turnaround times.”

The Azure network effect

That level of speed and agility sets Climavision up perfectly for the groundbreaking work that lies ahead. As the company continues deploying more radar, it needs more compute and scale to process and parse the staggering amounts of data. “Climavision is building a network of more than 200 radar installations across the United States,” says Goode. “With Azure HPC, we now have the means of moving the roughly 4.3 terabytes of data that each produces every month, for a total of around a half petabyte of data per month.”

Managing and processing data at those levels to get accurate forecast information to clients consistently is exactly what Climavision needs to fulfill its mission. “The radar information we’re bringing in and out of processing is a high-octane fuel source for all of our weather applications,” states Childs. “Speed is important. We need immediate access to radar data so we can run it on our large compute instances, produce a forecast, and create actionable information for our customers, and we get that with Azure HPC.”

Using the cloud to stay ahead of the weather

Climavision is now realizing the speed, reliability, and agility it needs. Historically, the compute necessary to handle and analyze data at this level was limited by long procurement cycles to add capacity and scale. Companies today can move much faster in scaling out their HPC environments. “Azure HPC gives us the speed we need to innovate quickly,” says Goode. “We were up and running with our GRO Global Forecast Model in five and a half months. That’s extraordinary!”

With its Azure capabilities, Climavision can stay ahead of the weather by spinning up nodes on demand, prototyping and developing faster, conducting data validation far more quickly, and configuring new models more rapidly and easily. “We might not be able to change the weather,” says Goode, “but the groundbreaking precision and resolution that our technology provides, built on essential Azure resources, means we can help our clients better prepare for whatever is coming.”

Find out more about Climavision on Twitter and LinkedIn. #AzureHPCAI