Transforming how Microsoft uses Microsoft Azure ExpressRoute

Our Microsoft Digital Employee Experience (MDEE) team manages one of the largest Microsoft Azure enterprise environments.

Migrating our virtual machine resources into Microsoft Azure necessitated significant connectivity between our on-premises networks and Azure. We have many Microsoft Azure ExpressRoute circuits connecting our on-premises resources to our Azure environment, and to ensure proper management of a network environment this large, we’ve moved our Microsoft Azure networking environment to a hub and spoke shared circuit model. This gives uniform access for our internal business groups to ExpressRoute and other Azure networking components, and it also allows us to centrally manage and monitor how we provision our Azure networking.

Understanding Microsoft Azure ExpressRoute usage at Microsoft

Microsoft Azure ExpressRoute is the default connectivity method we use for connecting Microsoft Azure to our internal networks. It’s the best solution in Azure for our large-scale network environment, compared to alternatives such as point-to-site or site-to-site virtual networks. We use ExpressRoute to connect Azure to datacenters, corporate headquarters, and regional offices across many different business groups. We use ExpressRoute for two primary purposes:

To connect internal or partner on-premises resources to a Microsoft Azure network containing virtual machines—typically to form a contiguous, isolated network space between Azure and an on-premises network.
To provide dedicated and measurable bandwidth between on-premises resources and Microsoft Azure components.

Rethinking Microsoft Azure ExpressRoute business group subscriptions

When we first began our Microsoft Azure services, Microsoft Azure ExpressRoute connections were provisioned at the request of our business groups. We perform the ExpressRoute provisioning and assign IP address ranges to accompany the ExpressRoute circuit within the Azure subscription that belongs to the business group or partner organization. Although this approach has the benefit of leaving the ExpressRoute ownership in the hands of the business group that owns the subscription, it does have some management challenges:

It doesn’t scale well. As more and more of our business teams began to embrace Microsoft Azure, it became quickly clear that a one-to-one relationship between Microsoft Azure ExpressRoute circuits and subscriptions wasn’t going to scale.
A business team might initially request an IP address range with a Microsoft Azure ExpressRoute circuit based on their current use of resources, and then require more at a later date. Reconfiguring the ExpressRoute circuit to add a larger IP range requires additional work to add or change hardware.
It requires a central account in each subscription to manage the circuits. Multiple accounts are complex and time-consuming to manage.
There’s not a single point of reference for where our Microsoft Azure ExpressRoute circuits have been provisioned.
Microsoft Azure ExpressRoute circuits must be provisioned individually. Provisioning tasks don’t only include creating the ExpressRoute circuit—it also includes allocating IP addresses, configuring routers, and other network-related tasks. Having to configure routers for every request quickly started to impact the amount of time required for creating connections.

Using the provider subscription model

The provider subscription model is intended to simplify the provisioning and management of Microsoft Azure ExpressRoute circuits and virtual networks that can leverage them. In this model, we use a single shared provider subscription that owns every physical ExpressRoute circuit. These ExpressRoute circuits are then used by other consumer subscriptions that belong to our business groups by connecting virtual network to virtual network across subscriptions.

Figure of the ExpressRoute shared circuit model that shows a series of objects connected to each other from left to right. — The Microsoft Azure ExpressRoute hub and spoke shared circuit model.

Implementing a hub shared provider subscription

To better manage the virtual networks for our Microsoft Azure ExpressRoute environment, we created shared provider subscriptions to specifically manage the ExpressRoute circuits. These provider subscriptions are owned by our central MDEE Infrastructure team. This team provides services out to the wider business units within MDEE and the larger corporate teams within all of Microsoft.

The process for linking a Microsoft Azure ExpressRoute circuit to an external Microsoft Azure consumer subscription is a straightforward process and, after it’s complete, our business groups can use the ExpressRoute connectivity in their own Azure environments. When we organize the ExpressRoute circuits this way, we can share connectivity between multiple business groups who may be using different subscriptions. This is beneficial when a single business group might not require the full amount of available bandwidth associated with an ExpressRoute circuit. It also adds greater flexibility to increase the number of IPs required later.

In the graphic below, MSIT-CORP-ERPROVIDER-1 represents the Microsoft Azure subscription that is designated as the provider subscription for owning Microsoft Azure ExpressRoute circuits. Within the subscription are Azure resource groups, named for Azure regions, into which are placed the ExpressRoute circuits that are then used by the consumer subscriptions that belong to our business units.

Figure shows the Microsoft IT implementation of the provider subscription model for ExpressRoute networks. — The MDEE implementation of the provider subscription model for Microsoft Azure ExpressRoute networks.

The provider subscription has a simple setup. For each region where we’re going to connect a Microsoft Azure ExpressRoute for connectivity, we create a resource group. This resource group owns the resources associated with that virtual network connectivity. We work with our central IP management team to allocate a large IP range for each circuit, from which smaller IP address ranges are assigned. As a particular region grows, we might need an additional circuit added. The table below provides an example of our configuration and naming conventions.

Subscription name	Resource group name	ExpressRoute circuit name
MSIT-Corp-ERProvider-01	WestUS	MSIT-CORP-WUS-Ckt-1
MSIT-Corp-ERProvider-01	WestUS	MSIT-CORP-WUS-Ckt-2
MSIT-Corp-ERProvider-01	NorthEurope	MSIT-CORP-NEU-Ckt-1
MSIT-Corp-ERProvider-01	NorthEurope	MSIT-CORP-WUS-Ckt-2
MSIT-Corp-ERProvider-01	SouthEastAsia	MSIT-CORP-SEA-Ckt-1
MSIT-Corp-ERProvider-01	SouthEastAsia	MSIT-CORP-SEA-Ckt-2
MSIT-Corp-ERProvider-01	SouthCentralUS	MSIT-CORP-SCUS-Ckt-1
MSIT-Corp-ERProvider-01	SouthCentralUS	MSIT-CORP-SCUS-Ckt-2
MSIT-Corp-ERProvider-01	CentralUS	MSIT-CORP-CUS-Ckt-1
MSIT-Corp-ERProvider-01	CentralUS	MSIT-CORP-CUS-Ckt-2
MSIT-Corp-ERProvider-01	CentralUS	MSIT-CORP-CUS-Ckt-3
MSIT-Corp-ERProvider-01	CentralUS	MSIT-CORP-CUS-Ckt-4
MSIT-Corp-ERProvider-01	EastUS2	MSIT-CORP-EUS2-Ckt-1
MSIT-Corp-ERProvider-01	EastUS2	MSIT-CORP-EUS2-Ckt-2
MSIT-Corp-ERProvider-01	WestCentralUS	MSIT-CORP-WCUS-Ckt-1
MSIT-Corp-ERProvider-01	WestCentralUS	MSIT-CORP-WCUS-Ckt-2
MSIT-Corp-ERProvider-01	WestCentralUS	MSIT-CORP-WCUS-Ckt-3
MSIT-Corp-ERProvider-01	WestCentralUS	MSIT-CORP-WCUS-Ckt-4
MSIT-Corp-ERProvider-01	WestUS2	MSIT-CORP-WUS2-Ckt-1
MSIT-Corp-ERProvider-01	WestUS2	MSIT-CORP-WUS2-Ckt-2
MSIT-Corp-ERProvider-01	WestUS2	MSIT-CORP-WUS2-Ckt-3
MSIT-Corp-ERProvider-01	WestUS2	MSIT-CORP-WUS2-Ckt-4
MSIT-Corp-ERProvider-01	WestUS2	MSIT-CORP-WUS2-Ckt-5

Naming conventions for Microsoft Azure subscriptions, resource groups, and Microsoft Azure ExpressRoute circuits.

As you can see from the table, we have Microsoft Azure ExpressRoute circuits configured across various parts of the United States, as well as Europe and Southeast Asia. We used a defined naming convention for subscriptions, resource groups, and ExpressRoute circuits—this gives us a standardized environment and simplifies ExpressRoute maintenance and usage. All of these resources can be managed within the same centralized shared provider subscription. As you can see, some regions have more circuits configured for them than others, reflecting our ability to use this model to grow based on need. As might be expected, the WestUS2 region has the most circuits (five) because it’s located near the main Microsoft campus in Redmond, Washington.

Getting our internal customers connected

As part of pivoting our Microsoft Azure ExpressRoute services for our internal customers, we had to provide a way to request an ExpressRoute connection for their subscription. This was initially implemented by creating a website that could gather pertinent information from them to implement the request and then pass it on for use in the implementation process. The graphic below shows the new network form on our internal website that collects the following information about the request:

Request name: Plain text to identify the request, like “New Finance IT ExpressRoute Request.”
Microsoft Azure subscription: The Subscription ID (GUID) for the subscription to connect.
Network type: Drop down to pick between our corporate network and our perimeter network.
Region: Choose the region they need for the virtual network connection.
Network size: /26 or a /27 virtual network. Customers that require something larger can contact us to request an exception.
Service name: The service associated as defined within our Microsoft Azure service hierarchy.
Associated users: We required at least two named contacts for notifications, follow-up, and to contact if there are any concerns regarding the configuration. Our security team will also use these contacts as required for detected compliance issues.

Figure shows a screen capture of the New Network creation page for requesting ExpressRoute services from a web form. — The new network creation page for requesting Microsoft Azure ExpressRoute services.

After we have this information, we use some simple automation that creates a resource group within the internal customer’s subscription to host the new virtual network resources. We create a virtual network of the appropriate size in the consumer subscription and link it with the appropriate Microsoft Azure ExpressRoute circuit in the provider subscription. This is now a spoke in the hub and spoke model.

The example in the graphic below shows the ERNETWORK resource group, which is created in the consumer subscription. In this example, a virtual network named MSIT-CORP-WUS-VNET-1 is also created in the ERNETWORK resource group, along with other network resources, such as a public IP address and a gateway. As the circuit owner, we create an authorization key in the provider subscription that can be used by the consumer subscription to access the Microsoft Azure ExpressRoute circuit. Finally, the authorization key is used to create a connection named MSIT-CORP-WUS-Conn-1 in the consumer subscription to link the gateway in the consumer subscription to the ExpressRoute circuit in the provider subscription.

An example of ExpressRoute network infrastructure in Azure. — An example of Microsoft Azure ExpressRoute network infrastructure in Microsoft Azure.

Our goal with the provider subscription model is multilayered, but it ultimately comes down to efficiency in management. By hosting Microsoft Azure ExpressRoute circuits in a single subscription, we’ve experienced several benefits to reduce the amount of Microsoft Azure peering overhead, enable faster ExpressRoute circuit provisioning, reduce routing overhead, and optimize our use of finite routable IP space. We’ve also found several benefits:

Microsoft Azure ExpressRoute circuits are hosted in fewer subscriptions, which provides centralized management and reduces sprawl.
Dedicated bandwidth can be shared between multiple subscriptions, reducing costs and overhead.
Provisioning is simplified. There’s one place to create Microsoft Azure ExpressRoute circuits, and having fewer individually created ExpressRoute connections reduces the network configuration overhead in modifying network routes and configuring IP subnets.
The model can easily be used by vendors, partners, and other third parties without the requirement to exchange account authentication information.
Troubleshooting and management is simplified because our central IT team owns and has access to the subscription where the physical Microsoft Azure ExpressRoute circuit is hosted.
We can allocate virtual networks in consumer subscriptions in a round-robin fashion to maintain load balancing throughout the network environment.
The relationship between MDEE and the business groups is more accurately reflected. As the group responsible for network configuration and management, we have primary control over how our network is being used by the business groups.