Businesses predict weather impact using cloud-based machine learning

Defend business against the weather

It’s tornado season in the midwestern United States. The supply chain manager for a big-box retailer needs to make daily decisions about delivery routes to the stores in the region. The operations team needs to make decisions about store safety, staffing, and inventory. The marketing team can boost customer loyalty and profits by offering discounted prices on roofing materials at the right stores and the right time.

To make the best decisions, they all need data—about their company’s previous-year sales and about the weather. That’s where AccuWeather comes in.

Based in State College, Pennsylvania, AccuWeather is a leading global provider of weather forecasts. It provides minute-by-minute forecasts for 2.3 million locations around the world for more than 100 parameters—including temperature, humidity, rain, snow, and ice—for every hour over the next 90 days and every minute for the next two hours.

AccuWeather gathers hundreds of thousands of real-time weather observations from land, ships, aircraft, satellites, and radar and crunches that data using its patented AccuWeather Forecast Engine. More than 180,000 websites, 200 television stations, 900 radio stations, and 600 newspapers feature AccuWeather forecasts. In total, nearly two billion people worldwide rely on AccuWeather to help them plan their lives, protect their businesses, and stay safe.

Even with all this weather data available in just about any format you can think of—as APIs that can be plugged into applications and through AccuWeather.com, AccuWeather apps, and the AccuWeather Network cable channel—businesses wanted more. With increasing unpredictability in weather and so much on the line, businesses asked AccuWeather for custom predictions of how weather would affect them.

“For a while, we provided this service on an ad hoc consultative basis, but it was very labor intensive,” says Rosemary Yeilding Radich, Director, Data Science, at AccuWeather. “Customers would share their sales history data with us, and we’d have a data scientist clean the data and create a custom model. We talked about how we could scale this service—have customers upload their data online and generate an automated prediction—but we feared that the analytical tools we had at the time wouldn’t allow us to provide them with a high-quality prediction.”

Transform weather prediction in the cloud

AccuWeather decided that this was a perfect problem for the cloud. The company runs its crown jewels, the Forecast Engine and its artificial intelligence (AI) algorithms, in its own datacenters, but AccuWeather started pushing workloads into public clouds around 2012.

“The impetus for moving to the cloud was better scalability and resilience, which we need for delivering timely information such as tornado warnings to our customers,” says Chris Patti, Chief Technology Officer at AccuWeather. “The cloud gives us better, more cost-effective disaster recovery than we can engineer on-premises and allows us to site our data as close to customers as possible.”

AccuWeather has made Microsoft Azure its preferred cloud platform. “Azure has a solid infrastructure as a service offering, and it also has a rich and rapidly advancing platform as a service portfolio, including big data, machine learning, and AI capabilities,” Patti says. “These are important to the data science side of our business.”

The company first moved its API business to Azure. AccuWeather sells a host of weather-related APIs that news outlets and other companies plug into their enterprise resource planning, customer relationship management, and other business applications. The company receives more than 40 billion API calls a day for weather information, and Azure provides a more scalable, cost-effective way to manage this service.

Next the company moved its big data storage and processing, using services such as Azure Blob storage, Azure Data Factory, and Azure SQL Database.

When AccuWeather set out to build an automated, scalable weather prediction service for business customers, it turned to Azure as well. To create highly accurate forecasts, Radich’s team needed sophisticated machine learning tools that it could customize using R and Python code.

“Azure really stands out from other clouds by providing out-of-the-box machine learning capabilities that are powerful yet customizable using R and Python,” Radich says. “It’s very important to our data scientists to have a cloud platform that plays well with open source, and that was one of the things that attracted us to Azure.”

David Mitchell, Vice President of Digital Media and Emerging Platforms at AccuWeather, adds, “We also really appreciate how closely Microsoft works with us. For instance, Microsoft shared how its other customers were using Azure Machine Learning services. That openness—in conjunction with the ability to scale our software in terms of number of processors, geographical reach, data protection, and other dimensions—was critical.”

To help AccuWeather accelerate the development of its new service, Microsoft connected it with Dynamic Data, a member of the Microsoft Partner Network that developed the user interface. “Microsoft focused on making us successful and directed us to resources so that we didn’t have to build everything ourselves,” Radich says. “Dynamic Data was like a member of our team and really helped us gain development speed.”

Launch AccuWeather D³ Data-Driven Decisions

In May 2017, AccuWeather launched AccuWeather D³ Express, a cloud-based analytics product that quantifies the impact of disruptive weather on a business using a rating of 1 (insignificant) to 10 (extreme). Businesses pay a monthly subscription based on the number of US zip codes they need “scored” for weather impact.

For example, a supply chain manager alerted about potential disruptive weather could quickly see the big picture of which routes and stores would be impacted. With ample lead time to take appropriate actions, he could alter delivery routes to ensure timely completion. Marketing teams could increase revenue through targeted product sales, and operations teams could reduce expenses from understaffing, overstaffing, and potential stock-outs.

D³ Express combines several Azure services to move, store, and manage data and to present data to the web or API client; these services include Blob storage, Azure App Service, and Azure API Management.

As soon as D³ Express entered the market, AccuWeather began work on the next version, D³ Advanced, which delivers even more customized predictions using machine learning. Businesses upload historical sales data for specific products and minutes later get spot analytics on a dashboard showing how sales of those products will be affected by specific weather conditions—wind, rain, humidity, and others. D³ Advanced is flexible enough so that the model works well whether a customer uploads sales data on all its stores in a region, the sales data for just one store, or the sales data for just one SKU.

AccuWeather built D³ Advanced using Machine Learning and other Azure data services. See the technical addendum for architectural details.

“Businesses can’t control the weather, but by using D³ Advanced, they can maximize their return on weather-driven opportunities and minimize losses,” Radich says. “D³ Advanced analyzes the relationship between historic and emerging weather patterns and a business’s key performance indicators. Analyzing this data identifies opportunities to help a business not only protect its bottom line but make the most of predicted weather conditions and anomalies across all business locations.”

Let’s say a forecast calls for snow. Should a supply chain manager stock up on snow tires or chains? Instead of guessing, or ordering both, she sees from D³ Advanced that last year, customers bought more chains than snow tires in a two-to-one ratio under similar conditions. So she orders twice as many chains as snow tires. But just as each region’s forecast can vary, consumer patterns related to snow can as well. AccuWeather D³ Advanced can let businesses see how consumers in one region will need boots and snow tires to get around, while in another region, people will stock up on cocoa, wine, and board games for a few days inside.

Or, the director of operations for a sunglasses manufacturer uses D³ to see weather conditions across the country at a glance and notices that a streak of hot weather is expected in the US South Atlantic region. Historical data shows a twofold boost in sales during similar weather in previous years, so he increases production levels and gets a jump on the competition.

Innovate faster with data science services on demand

By developing D³ Advanced in Azure, AccuWeather was able to get to market faster than it could have if it had developed its own machine learning services. “With Azure, we have access to sophisticated data science services in a real-time, on-demand basis, which helps us innovate faster,” Patti says. “Setting up things like Hadoop clusters and networking is push-button easy in Azure. We haven’t had to hire people with that expertise.”

Radich says that the company’s data scientists love the visualizations that Machine Learning provides when building a model—bar and line charts and scatter graphs that help them better understand the data and reduce overfitting, so algorithms are flexible enough to be reliable.

Instead of developing a one-off model for every customer, AccuWeather data scientists can focus on creating great foundational technology that delivers a scalable solution for many customers. This frees up time to develop new products and provide even more tailored consulting services.

“We see data and analytics as a huge growth area for us, and we’ve made it a major initiative to increase our ability to provide more insights from D³,” Radich says. “By using Azure Machine Learning, we can meet a variety of customer needs, learn from all the analytics, and put that learning into new products and services to improve our offerings.”

Moving forward, AccuWeather will conduct all its data and analytics product development and hosting in Azure. And the company is moving all its products out of AccuWeather datacenters and into the cloud. “Some of our forecast systems are still on-premises, but we plan to move all of them to the cloud, along with data ingestion, weather data processing, you name it. We’re transforming our business with Azure.”

Technical addendum: elegant data mashup in Azure

AccuWeather D³ Advanced provides actionable insights based on an automated comparison of a customer’s unique sales history and AccuWeather historical weather data. AccuWeather supplies historical weather from its datacenters, using a Python script running on a Linux-based server to periodically load new files daily. This data is then loaded into Azure SQL Database using the bulk copy protocol (BCP).

Customers upload their historical sales data using the D³ Advanced Historical Sales Upload Form that runs in an Azure App Service web app. SQL Database serves as the data store for both AccuWeather weather data and the customer’s historical sales data, providing aggregation and cleansing.

AccuWeather uses Azure Machine Learning Studio to train models for each subscriber/customer based on historical weather and their provided sales data. Azure Data Factory generates updated sales forecasts based on changing weather forecasts and stores data in SQL Database for the D³ Advanced Analytics Visualizations.

Find out more about AccuWeather on Twitter, Facebook, and YouTube.