Global energy giant bp is transforming into an integrated multi-source energy provider. To speed innovations in support of this initiative, the company is working to democratize data ownership, enhance and streamline data sharing, and optimize the flexibility of its platform by leveraging the Microsoft Intelligent Data Platform for a new multicloud modular data mesh architecture.
“Modularity is at the heart of bp’s digital approach to drive an integrated set of world-class customer experiences, reduce the time to value for new products, and enable bp to create new businesses.”
Abeth Go, Vice President for Data & Analytic Platforms, bp
With operations in over 80 countries and over 164 billion USD in revenue generated in 2021 alone, bp is one of the world’s largest energy companies. Bp has a bold vision for bringing greater diversity and integration to its energy sources. The company’s vast data platform has a vital role to play in bringing this vision to reality, by enabling data sharing and data management across its many business units faster and more effectively than ever. “As our strategy pivots to integrated energy, there's a strong focus on delivering innovative solutions to our customers,” explains Abeth Go, Vice President for Data & Analytics Platforms at bp. “We need to be able to mobilize very swiftly, especially when we're entering new markets. It's important that we have a modular operating model so that we can bring multidisciplinary teams together in a shared ecosystem of data assets and data products that spans the organization.”
In the past, bp’s digital production and operations relied chiefly on large data lakes that served multiple business units. As new products and offerings drew increased focus, priorities shifted toward self-service analytics, streamlined deployment times, and multi-environment integration. Enterprise-wide data governance and security were also chief priorities during this time of rapid change.
bp reached out to Microsoft, already its long-time cloud provider, for insights into how it might adapt its data lakes to a more modern, distributed architecture. Both companies dedicated teams to the creation of the solution, which would take shape as a modular data mesh based on the Microsoft Intelligent Data Platform and incorporate Azure Databricks, Azure Synapse Analytics, Microsoft Purview, and Power BI. The Microsoft Intelligent Data Platform is the leading cloud data platform that fully integrates databases, analytics, and governance. This seamless data platform empowers organizations to invest more time creating value, rather than integrating and managing their data estate.
Envisioning a new kind of platform
The process began with exploratory sessions, including regular scoping and envisioning meetings. Liam Donohoe, Principal Architect, Data & Analytics Platforms at bp, presented his team’s work on the modular reuse of data products to create an ecosystem of data across the company’s federated data platform environments. “That’s what brought home the potential power of a data mesh implementation at bp,” says Donohoe. “The big successes we'd had in bringing the data from our legacy data lake in Azure and our primary geospatial system together really convinced the team that this was the way to move forward.”
bp’s data platform team worked to understand the technology, organization, and data culture of each business unit. Many at bp, especially those in less technology-focused positions, needed to understand how a data mesh architecture would help them speed innovation.
Microsoft initiated several workshops with data leaders and owners from each business group within bp, plus representatives from its digital security, identity, networking, and infrastructure teams. Through a dedicated enablement team, Microsoft launched an Enterprise Skills Initiative, which bp quickly embraced. bp also began a variety of skilling workshops, including an internally created champions program and multiple certification initiatives from Microsoft Learn that helped scale the people-driven adoption of the solution organization-wide. Internal bp learning programs were used to bolster these certification programs, and by tracking the overall number of Microsoft Certifications held throughout the company, the data platform team could see how much progress was being made.
Building the bp Data Hub
These foundational steps took approximately nine months and were critical to producing bp’s first data mesh framework. “We collaborated and partnered with Microsoft on what our data platform would look like in Azure,” says Donohoe. “This ended up being built in Microsoft's own lab environments to help identify specific implementation details, particularly around the separation of control and data planes and how we establish data sharing across the various data product environments.”
Within their Data Hub—bp’s core data ecosystem—the company is applying a variety of data mesh principles in a multicloud, modular structure. The Data Hub realizes the primary data mesh cells of the wider data mesh, each of which is its own Azure hosting region.
Each data mesh cell contains a data management and governance hub adapted from Azure data management landing zone architecture. Services like Azure Firewall, Azure Synapse Analytics Private Link Hub, Azure Data Factory, and Microsoft Purview help assure data governance and maintain secure connections between the multiple “data nodes”—built from Azure data landing zones. “Each data node is organized around vertical business groups and natural groupings of teams, based on the sorts of data products they use,” explains Donohoe. These nodes form a dedicated data network optimized for federated data sharing and the convergence of operational and analytical workloads. The nodes contain data domains and node governance that integrates with the central data management hub at the cell level.
A multicloud solution, the bp Data Hub is designed to integrate the entire data value chain and provide a consistent, persona-driven data experience across the company’s Microsoft and third-party cloud environments. With Microsoft Purview, bp is working to provide unified multicloud data governance through automated data discovery, sensitive data classification, and end-to-end data lineage. Data sourced from anywhere within bp’s data platform can consequently be verified, qualified, highly reliable, and secure by design. This will allow data sourced from users to ensure that their analytical and commercial uses for the data are as verified and qualified possible.
Larger volumes of data sourced from across the organization will soon be readily available with far fewer restrictions, allowing for the creation of an ecosystem of reusable companywide data products. It will also help deliver even greater benefit from the company’s already world-leading Power BI implementation. The breadth and depth of self-service insights bp will soon visualize through Power BI will help the company see opportunity where others might not and bring greater refinement to even the most optimized processes.
bp already has numerous data domains operating data cells, each aligned to specific teams dedicated to progressing data products through their life cycles. “Essentially, these data cells represent resource groups or team workspaces that assemble all the various resources and experiences that each team needs to deliver a particular type of data product,” explains Donohoe. Through this framework, data models, knowledge models, analytics, and digital twin solutions are already in the works.
Delivering solutions that will power the future
A major benefit of bp’s data platform is its modularity. Each new digital service the company creates is designed to consolidate the activities, technologies, services, and resources required to deliver a specific business outcome, and can also be used over and over by many different teams in service of multiple products while preserving data ownership at its point of origin. “Modularity is at the heart of bp’s digital approach to drive an integrated set of world class customer experiences, reduce the time to value for new products, and enable bp to create new businesses,” says Go.
Now in its second iteration, the bp Data Hub is being expanded to focus on the potential benefits it lends to edge computing and digital twin technology. “bp and Microsoft have been working together on innovative solutions for years,” says Go. “When we brought this idea of a data mesh implementation to Microsoft, experts there welcomed the opportunity. Now we’re working together to take on the challenge of transforming how data products can be managed and delivered in an innovative, modular, and self-service manner.”
For organizations considering a data mesh similar to bp’s, Go has the following advice: “Don’t start from scratch. Leverage what’s already working well for you. Also, don’t underestimate the impact on people's ways of working; back up your new platform with a flexible operating model. Establish regular and transparent communication, build partnerships within your organization, and you’ll have a dedicated team at your back when you start reaping the rewards of a modular data mesh capable of driving your business forward.”
“We need to be able to mobilize very swiftly, especially when we're entering new markets. It's important that we have a modular operating model so that we can bring multidisciplinary teams together in a shared ecosystem of data assets and data products that spans the organization.”
Abeth Go, Vice President for Data & Analytics Platforms, bp
Follow Microsoft