In the energy industry, Shell manages everything from wells to retail gas stations—44,000 of them. The company works hard to ensure the safety of service champions and customers at its retail sites. Shell is piloting a new cloud-based, deep learning solution built on Microsoft Azure. The solution uses closed-circuit camera footage and Internet of Things technology to automatically identify safety hazards and alert service champions so they can quickly respond and eliminate potential problems.
“With Azure, we not only get an intelligent cloud, we get an intelligent edge, which helps us automatically identify and respond to safety hazards in near real time.”
Daniel Jeavons, General Manager for Data Science, Shell
The views expressed by Shell employees in this customer story are a testimonial based on current experience, and not an endorsement. Others may not have the same experience/results.
It can see and predict the future in ways that human eyes can’t. It’s called “machine vision,” and it uses imaging technology, combined with image processing and analysis, to guide machine action and human decisions in a commercial or industrial setting.
Shell is now piloting a new machine vision system at retail locations. By applying advanced artificial intelligence techniques, the system can automatically predict and detect unsafe actions in a shop or forecourt and alert staff so that they can intervene. It’s being applied with an eye toward Goal Zero—Shell’s ambition to achieve no harm and no leaks across all of its operations.
Here’s how Shell envisions this system at work. Imagine a sunny day—a customer pulls his car into a Shell gas station for a fill-up. He pops the lid on his gas tank and starts refueling. While he’s waiting, he reaches into his pocket for a cigarette and lights it, oblivious to the safety signs that indicate how dangerous this act can be.
An onsite camera captures footage of the situation, and image processing running on the device via Microsoft Azure IoT Edge identifies the potential safety issue. It relays the image through Azure IoT Hub to the cloud, where deep learning and artificial intelligence (AI) models running via Azure Databricks detect patterns that match a cigarette. This information is immediately sent to the gas station, where it pops up as an alert on the computer dashboard of the station manager—one of Shell’s service champions. Before the customer can take another drag on his cigarette, the manager disables the pump to avoid any potential incident.
After the customer puts out the cigarette, he is able to finish refueling, and it’s back to business as usual.
Putting safety first
Safety. It’s at the heart of everything Shell does. From exploration to extraction and all the way to retail vending, Shell prioritizes the safety of its employees, contractors, partners and customers, and the environment. The company makes increasing use of technology to enable this mission, transforming the way it does business and the way it interacts with all the people who make that business possible.
Headquartered in The Hague, the Netherlands, Shell is a major global company. Shell is the number-one mobility retailer in the world, serving 30 million customers a day across a network of 44,000 stations in more than 75 countries. Shell strives to provide a safe and welcoming environment—for both customers and service champions—at every one of its retail sites. This includes mitigating risks of vehicle collisions, smoking, and improper fueling behaviors.
Shell trains its service champions to detect risks and intervene, and it aims to make that easier to do on a 24-hour basis with an innovative new project known as Video Analytics for Downstream Retail (VADR). Using machine vision technology, combined with automated image processing and analysis, VADR adds an extra level of safety for people and the environment. “Things like smoking pose really big risks,” says Daniel Jeavons, General Manager for Data Science at Shell. “We want to use digital technology to be able to respond quickly and correctly to prevent dangerous scenarios. We consider that not just good business, but part of our responsibility as a member of the communities we’re part of.”
Finding a cloud platform for the fastest possible response
The VADR project stems from an effort by Shell to reduce risks at retail sites while simultaneously increasing operational effectiveness and efficiency. Shell selected Microsoft Azure as the cloud platform for VADR. Not only did Azure offer the low latency Shell needed to enable fast, actionable insights, but the company could take advantage of powerful Azure services like Azure Databricks and Visual Studio Team Services, plus easy access to GPUs (graphics processing units).
“We think that the investments Microsoft is making in the area of machine vision and AI position it very well as a technology partner,” says Jeavons. “The availability of tools like Azure Databricks and GPU processing power in the cloud make it clear that Microsoft is thinking the same way we are, which is great. And because we already use Team Services, we can run an end-to-end DevOps pipeline straight into Azure with all the containerization and failover management we need at the scale we’re running. It even works great for AI models—something that traditional DevOps pipelines don’t support.”
Using the right tools for the job
To make its solution run as efficiently as possible, Shell deployed Azure IoT Edge so that its application logic would be as close as possible to the situations it analyzes, further reducing the time the system needs to make vital safety decisions. This is a significant improvement for Shell over systems in which all processing happens in the cloud.
“With Azure, we not only get an intelligent cloud, we get an intelligent edge, which helps us automatically identify and respond to safety hazards in near real time,” says Jeavons. “We have a set of containerized flows that include analytic models. We push them to the edge using Apache Kafka streaming—which is really good for managing large data sets—and the closed-circuit TV camera footage passes through the models, which extract the frames we need to identify potential safety risks.”
The VADR application then passes the frames up to Azure, where the application runs tightly trained deep learning models on them using Kafka, in conjunction with Azure Databricks running on Apache Spark. If the application detects a suspicious event, it triggers an alert in a web-based dashboard where a service champion will see it. Shell has found Azure Databricks to be a key element of the solution.
“We’re using compute-intensive learning models,” says Jeavons. “What we really like about Azure Databricks is that it runs on top of a very stable and mature installation of Spark. It also interacts closely with Kafka. We can easily scale up and retrain our models on a continuous basis and deal with our intensive computing needs. And because Azure Databricks is highly elastic, we get really powerful spin up/spin down capabilities, and our developers love its neat, elegant user interface.”
Shell appreciates the ability to work with open-source solutions within Azure. “The support for open-source software in Azure is crucial for us—we use Kafka and Spark, deep learning frameworks like TensorFlow, and machine vision solutions like OpenCV,” says Jeavons. “Our developers find it helpful to be able to pivot between Azure IoT Hub and Kafka for different parts of the solution. We like the ability to take the best of both worlds.”
“And we’re taking advantage of the Azure Databricks shared environment,” he continues. “We’ve made it our preferred collaboration platform, and it’s helping our data scientists and engineers share more and get to the next level of AI sophistication. When we look at the entire Azure platform, and the way all the pieces work together so well, it has definitely changed the game for us 100 percent. We’re using it to work in a true DevOps—or data ops, if you want to call it that—fashion, so we’re able to deploy models as code and push them into production in a continuous manner. That’s big.”
Embracing deep learning and expanding the solution
Shell is running a pilot deployment of VADR at retail stations in Thailand and Singapore, with plans to expand the project once those installations have proven successful. The experience with machine vision and AI has sparked a keen interest in further AI projects within Shell, and the company is starting a residency program for students who want to do AI work at Shell.
Throughout the development of VADR, Shell is going beyond just AI and machine learning and embracing deep learning. “These are sophisticated use cases we’re looking at, and they require more than just simple predictive models,” says Jeavons. “As a business, Shell has vast amounts of data, and deep learning with Azure is helping us make sense of it. We’re getting insights to trends in a completely new way, which enables better decision making.”
Shell aims to add VADR to an existing push to digitalize site management and transform the way service champions manage and run retail locations. Because the system captures video images of customers, the company is very sensitive to privacy issues, and it has worked hard to ensure that the solution fully complies with the General Data Protection Regulation (GDPR).
Most of the activities across Shell operations involve the management and control of physical assets—retail stations, tankers, lube plants, pipelines, refineries, and oil and gas platforms—so the VADR project and machine vision technologies have a wide range of potential applications within the company. These could include automated alerts for systems on remote offshore platforms in case of dangerous conditions or monitoring of vital assets, such as pipelines and wells, to prevent corrosion. “By automatically extracting information from photographs and video footage, we can better manage safety risks, keep our business running smoothly, and help our customers and employees feel secure and comfortable whenever they visit a Shell facility,” says Jeavons.
“We don’t put a dollar value on safety,” he continues. “We’re willing to invest in safety because it’s the right thing to do, and we believe that the financial metrics will follow from there. We’re very excited about the opportunities with Azure and AI for increasing safety, and we look forward to ongoing collaboration with strategic partners like Microsoft to develop these exciting solutions.”
Find out more about Shell on Twitter, Facebook, and LinkedIn.
“We’re taking advantage of the Azure Databricks shared environment—we’ve made it our preferred collaboration platform, and it’s helping our data scientists and engineers share more and get to the next level of AI sophistication.”
Daniel Jeavons, General Manager for Data Science, Shell
Follow Microsoft