SitePro accelerates green expansion using Azure Cache for Redis

Democratizing fluid management

Aaron Phillips, SitePro Co-CEO, is an expert in IoT and remote-control technologies for the oil and gas industry. He cofounded SitePro in 2012, and the company is now a leader in the IoT and intelligent control arena. Recently, Inc. magazine ranked SitePro in the top 500 fastest-growing private companies.

“We owe our success, our precision, and our attention to detail to the oil and gas industry,” Phillips says. “Along with our commitment to that industry, our mission has broadened. We have a social responsibility.”

SitePro created a single, cloud-based platform for oil and gas site management. The flagship product uses patented remote control technology to automate upstream and midstream controls and communications. It aggregates real-time streaming data from IoT sensors and controllers at facilities, pumps, valves, and wellheads. Site engineers can monitor and control every part of their field operations and solve issues from anywhere—enabling faster, safer, and more efficient operations.

Phillips and team saw an opportunity to bring their automated fluid management solution into new markets. In West Texas, where SitePro headquarters are located, drought has hit the agriculture market hard. The team began thinking about how to generalize its platform for a wider audience that needed precise and effective water management.

SitePro also began talking to municipalities to learn more about the challenges of sustainable, efficient water management. Dustin Brown, SitePro Director of Technology, noticed a trend. “The municipalities we talked to wanted to modernize their infrastructure but weren’t sure where to start,” he explains. “What we're replacing is very dated software in some cases, and that's a vulnerability.”

Municipalities also told SitePro that technical support had been a challenge. For-profit companies sell software solutions to public sector entities all the time—and then disappear. SitePro, on the other hand, aims to provide a best-in-class, ever-evolving platform and to stay current with modern technology, while delivering ongoing technical and field support.

SaaS on Azure turns out to be ‘a no-brainer’

Phillips and Brown looked for a solution in the software. They envisioned software as a service (SaaS)—a version of the SitePro platform that made it easy for customers to sign up and replace outdated, in-house solutions.

“Azure was a no-brainer,” Phillips claims. Originally, the company moved to Azure when its on-premises servers crashed many years ago. “I couldn't get the backups. I only had my laptop. So, I said, ‘I guess we're going to the cloud now.’ In one week, I got everything up in Azure. That speaks to the ease, and it's only gotten easier.”

The SitePro platform manages high volumes of unstructured data from onsite IoT sensors and other monitoring devices. At first, the team used a combination of back-end Azure data services to store the real-time and historical data, including three managed databases—Azure Cosmos DB, Azure SQL Database, and Azure Cache for Redis.

“We know, love, and have used Redis for years as a key component in our real-time IoT and intelligent control platform,” reports Phillips. So, when Microsoft announced a new partnership with Redis Labs in 2020, he and his team took note.

This partnership resulted in new service tiers of Azure Cache for Redis that gave SitePro a way to refine its data storage solution. Azure Cache for Redis already had robust offerings built on open-source Redis, but the new Enterprise tier included powerful features that enabled them to do even more.

Specifically, the Enterprise tier includes the new RedisTimeSeries module and support for multi-terabyte datasets. The added resilience and performance in the Enterprise tier was a boost to the underlying data infrastructure. According to Phillips, “Its scalability played a major role in the speed at which we have been able to cross over into other industries without experiencing the data congestion issues associated with significant growth in a real-time IoT environment.”

Azure Cache for Redis backs a scalable IoT architecture

In the SitePro architecture, multiple proprietary services and web jobs grab real-time and historical data from hundreds of thousands of sensors, import weather and other external data, monitor communications, build reports, and provide context for users—all based on the company’s patented intelligent remote-control technology. Sensor data is streamed to the SitePro edge API and ingested by the proprietary dataflow.

When the company’s analytics suite first launched, the SitePro platform ran dozens of web jobs and app services in the background to handle change feeds and to pipe data to Azure Cosmos DB. As a multi-model database, it supports all types of streaming data that SitePro captures—from key-value pairs to graphical weather data to column models—in one database. For example, a job might pull summary data representing 100,000 data points and store it in a separate Azure Cosmos DB repo “like a homemade time-series database,” Phillips notes.

This system became much more efficient with the release of the RedisTimeSeries module. It allows SitePro to orchestrate real-time and historical data services in Azure Cache for Redis. “I can pipe the raw logs into the RedisTimeSeries without the overhead of having to maintain the original services,” Phillips explains. “There is an insane difference in the footprint!”

For example, 100 gigabytes (GB) of raw logs in the former data lake are now only 1.5 to 11 GB of RedisTimeSeries data. “I'm not storing all the text—just the primitives,” Phillips adds. “It's not down-sampling. It's raw to raw. It was an easy decision.”

The switch to the Enterprise tier of Azure Cache for Redis and the RedisTimeSeries module simplified the architecture. Plus, it improved data quality. In the past, there might have been a gap in the timing of incoming logs. Now the overhead is gone, and the runtime costs for a larger footprint are about the same as before for the smaller size.

“We can do more with Redis by leveraging the RedisTimeSeries module to support data collection and analytics from hundreds of thousands of IoT sensors,” Brown points out.

A simple UI and other benefits

As SitePro has broadened its mission to serve more industries, its essential focus on ease of use and quick results hasn’t changed. This is a company that added an IT division specifically to assist customers who needed help upgrading their out-of-date operating systems.

“I want to find the shortest amount of time to do something,” Phillips says, “and that's exactly what this platform is for our customers.” For example, the SitePro web, mobile, and onsite interfaces provide what he calls “intelligent remote control.” When an operator uses the SitePro mobile app to open a valve remotely, the software uses AI to calculate potential outcomes and to mitigate potentially dangerous results, such as a spill or a health, safety, and environmental (HSE) incident. If an action has a high probability of a dangerous outcome, the software drops the user’s command and provides a full explanation of the potential danger to the user.

For ease of use like this in the field, the developers use Windows Presentation Foundation (WPF), an open-source, graphical user interface for Windows. WPF provides a simple and consistent way to bring data from the code to the UI layer through data binding. According to Phillips, that’s why he used WPF. “You can bind data, and the operating system handles it instead of you having to.”

Other Azure technologies help the company improve on costs. SitePro takes advantage of the updated autoscale capabilities in Azure Cosmos DB, including the ability to customize throughput (max RU/s) values. “We used to change something, and of course we forgot about it!” Phillips laughs. “Autoscale has saved us a lot of time and money.”

In addition, the change feed in Azure Cosmos DB supports a decision audit trail—a unique feature of the SitePro platform that logs every decision made by the site control software. The decision audit trail provides a persistent record of changes to a container in the order they occur.

“I love Azure Cosmos DB for storing highly variable models, like the onsite software decision audit trail,” Phillips notes. The audit trail provides vital insights for customers who need to know how the app’s logic arrived at a particular control decision, such as adjusting a valve or changing the speed of a pump. Each audit log explains the decision that was made and provides data about that decision, enabling customers to understand the control logic and to adjust it, if needed, to impact similar decisions in the future.

“The change feed allows us to store specific or interesting software audit logs in other locations as they come in, without interfering with the normal data flow,” he adds.

The data archive also revealed fresh insights after the company began feeding motion-capture streams from video cameras at customer sites into Azure Computer Vision APIs. “I had no idea there are so many loose dogs running around in the oil fields,” Phillips jokes. The Computer Vision project is currently in its beta phase. Its goal is to recognize unusual situations that can be linked to alerts for users.

A digital revolution for natural resources

SitePro is shifting the way public and private sector organizations think about modern operations, and Azure is at the heart of the change. Two years ago, SitePro was focused on oil and gas, but even then, Phillips and team were looking forward. They believe that with Azure, the company’s solutions can have as big an impact in other industries.

“If you think about it, we stress about the thought of losing one barrel of oil out of the millions of barrels of fluid we track every day,” Phillips muses. “To bring that level of precision into other spaces is huge. We have that ability, and we'd love to apply it.”

Today the company is continuing to grow while focusing on constant improvements to ease the burden of companies running large or complex systems. The new architecture of the SitePro solution has adapted the best from the original and has tweaked the back end to simplify the dataflow, creating an environment for smarter IoT decision-making.

“With the business growth in IoT, success always comes with more live and historical data,” Phillips concludes. “We want to make your life easier. Once you’re connected to our cloud infrastructure, the things you can do with it are wild.”