The awesome ugly truth about decentralizing operations at Microsoft with a DevOps model

Jul 8, 2021   |  

Moving to Microsoft Azure was an important first step for us to take at Microsoft, but to take full advantage of being on the cloud, we needed to embrace using the Microsoft Azure DevOps model.

So far in our series on Microsoft’s expedition to the cloud, we’ve unleashed Microsoft Azure by automating incident and change management, compromised with Microsoft Azure as we moved and monitored applications, and built a plane while flying it (figuratively … while managing hybrid solutions) on our journey to the cloud.

I think we can conclude that operationalizing the cloud at Microsoft has certainly not been an “all clear and blue skies” experience. Beyond the learnings, pitfalls, and compromises we’ve gone through, the advancement in agile culture and operational processes we’re already seeing has made it all worth it.

In my last blog post, I shared how our incident and change management systems have been evolving. And quite frankly, so have we as an organization. As our teams embrace the Microsoft Azure DevOps model more and more, we’re re-evaluating the day-to-day operations, services architecture, and service delivery model.

The awesomely ugly truth about this operation is that we worked on replicating this exact model, somehow expecting a more lean and agile operation, and not surprisingly, it didn’t quite work out that way … at first.

– Pete Apple, cloud services engineer, Microsoft Digital

[Read the rest of Microsoft’s move to the cloud series: Automating Microsoft Azure incident and change management on Microsoft’s move to the cloud, The learnings, pitfalls, and compromises of Microsoft’s expedition to the cloud, and Managing Microsoft Azure solutions on Microsoft’s expedition to the cloud.]

About seven years ago, our applications ran on-premises, and my team delivered a service that provided physical and virtual machines. The process for employees to acquire these resources was simple: Go to an order page, fill out information about your system and storage size, and get your machine after the designated service level agreement (SLA)—which was usually a few days for a virtual machine and a few weeks for a physical one, depending on inventory.

Apple sits at a chair in an open space in a Microsoft building.
“Transitioning our services from centralized management to decentralized ownership and management has been ugly … and awesome,” Apple says. “Moving to the Microsoft Azure DevOps model has helped us get past the ugly to more of the awesome.” (Photo by Jim Adams | Inside Track)

The awesomely ugly truth about this operation is that we worked on replicating this exact model, somehow expecting a more lean and agile operation, and not surprisingly, it didn’t quite work out that way … at first.

My team created a series of Microsoft Azure subscriptions owned by my team that connected back to our corporate network via VPN (eventually Microsoft Azure ExpressRoute), which uploaded a standard operating system image to build a virtual machine (VM). With the help of Microsoft Azure Resource Manager (when it became available, along with its more nuanced role-based access controls), we included additional options on the requisition form where application teams were able to order Microsoft Azure VMs.

This worked fine … for a while. Then we started to see problems. Employees wanted to manage their VMs directly in the Microsoft Azure portal, but they only had remote desktop access. My team was the owner of the subscriptions. Any change to a VM, like resizing or adding another disk, had to have a ticket for my team to deliver this service, while the customer (sometimes impatiently) waited for us to process requests and meet our ugly SLA. We also had some application teams wanting to use other types of Microsoft Azure resources beyond VMs.

What were we to do?

Transitioning our services from centralized management to decentralized ownership and management has been ugly … and awesome. Moving to the Microsoft Azure DevOps model has helped us get past the ugly to more of the awesome. In many ways, it’s even more transformative than the journey from on-premises to the cloud.

– Pete Apple, cloud services engineer, Microsoft Digital

In the name of our DevOps model, we decided to evolve once again and modify this service to provide a shared ownership model. Under this new model, we created business-unit owned Microsoft Azure subscriptions where application teams directly manage resources, resize, add disks, do maintenance, or whatever they needed to do to their machines to keep their employees working smarter, not harder. All the while, my team maintains governance and assistance as needed via Azure governance.

We started to decentralize Microsoft Azure subscriptions by each business unit service line so teams could apply appropriate roles to employees and flip to a true DevOps mode. My central team provided Microsoft Azure Resource Manager templates that enabled employees to build VMs themselves (no order form required!) and have their machines ready in under 30 minutes. Employees could also start creating platform as a service (PaaS) resources to modernize their applications and manage them day to day.

Transitioning our services from centralized management to decentralized ownership and management has been ugly … and awesome. Moving to the Microsoft Azure DevOps model has helped us get past the ugly to more of the awesome. In many ways, it’s even more transformative than the journey from on-premises to the cloud.

We now enable our teams to create and manage their own Microsoft Azure resources directly, while maintaining standards and governance guardrails. This is a true reflection of how we’re empowering every person in our organization to do more. Enabling customers to work smarter, not harder, is a wonderful thing.

I love my job.

Find out more about Microsoft’s Cloud Adoption Framework for Microsoft Azure documentation.

Learn more about how Microsoft evolved its operations and moved its IT infrastructure management to the cloud.

Tags: , ,