Microsoft Digital is using a centrally managed hub and spoke model to increase efficiency, simplify management, and improve network performance for it’s Azure ExpressRoute environment.

EXPLORE RELATED CONTENT

Microsoft Digital the organization that powers, protects, and transforms Microsoft manages one of the largest Azure enterprise environments. We’ve been migrating workloads to Azure and creating new workloads in Azure more and more each year. As we’ve migrated many virtual machine resources directly into Azure, the transition has necessitated significant connectivity between our on-premises networks and Azure. We have a large number of ExpressRoute circuits connecting our on-premises resources to our Azure environment, and that number continues to grow quickly. To ensure proper management of a network environment this large, we’ve moved our Azure networking environment to a hub and spoke shared circuit model—this gives uniform access for our internal business groups to ExpressRoute and other Azure networking components, and it also allows us to centrally manage and monitor how we provision Azure networking.

Understanding ExpressRoute usage at Microsoft

ExpressRoute is the default connectivity method we use for connecting Azure to our internal networks. It’s the best solution in Azure for our large-scale network environment, compared to alternatives such as point-to-site or site-to-site virtual networks. We use ExpressRoute to connect Azure to datacenters, corporate headquarters, and regional offices across many different business groups. We use ExpressRoute for two primary purposes:

  • To connect internal or partner on-premises resources to an Azure network containing virtual machines—typically to form a contiguous, isolated network space between Azure and an on-premises network.
  • To provide dedicated and measurable bandwidth between on-premises resources and Azure components.

Difficulties with ExpressRoute hosted in internal business group subscriptions

When we first began our Azure services, ExpressRoute connections had been provisioned at the request of our business groups. We perform the ExpressRoute provisioning and assign IP address ranges to accompany the ExpressRoute circuit within the Azure subscription that belongs to the business group or partner organization. Although this approach has the benefit of leaving the ExpressRoute ownership in the hands of the business group that owns the subscription, it does have some management issues:

  • It doesn’t scale well. As more and more business teams began to embrace Azure, it became quickly clear that a one-to-one relationship between ExpressRoute circuits and subscriptions wasn’t going to scale.
  • A business team might initially request an IP address range with an ExpressRoute circuit based on their current use of resources, and then require more at a later date. Reconfiguring the ExpressRoute circuit to add a larger IP range requires additional work to add or change hardware.
  • It requires a central account in each subscription to manage the circuits. Multiple accounts are complex and time-consuming to manage.
  • There’s not a single point of reference for where our ExpressRoute circuits have been provisioned. 
  • ExpressRoute circuits must be provisioned individually. Provisioning tasks don’t only include creating the ExpressRoute circuit—it also includes allocating IP addresses, configuring routers, and other network-related tasks. Having to configure routers for every request quickly started to impact the amount of time required for creating connections.

Using the provider subscription model

The provider subscription model is intended to simplify the provisioning and management of ExpressRoute circuits and Virtual Networks that can leverage them. In this model, we use a single shared provider subscription that owns every physical ExpressRoute circuit. These ExpressRoute circuits are then used by other consumer subscriptions that belong to our business groups by connecting Virtual Network to Virtual Network across subscriptions.

A series of objects connected to each other from left to right that illustrate the Microsoft IT ExpressRoute design 1 to many mapping.A series of objects connected to each other from left to right that illustrate the Microsoft IT ExpressRoute design 1 to many mapping.

Figure 1. The ExpressRoute hub and spoke shared circuit model

Implementing a hub shared provider subscription

To better manage the virtual networks for our ExpressRoute environment, we created shared provider subscriptions to specifically manage the ExpressRoute circuits. These provider subscriptions are owned by our central Microsoft Digital Infrastructure team. This team provides services out to the wider business units within Microsoft Digital and the larger corporate teams within all of Microsoft. 

The process for linking an ExpressRoute circuit to an external Azure consumer subscription is a straightforward process and, after it’s complete, the business group can use the ExpressRoute connectivity in their own Azure environment. When we organize the ExpressRoute circuits this way, we can share connectivity between multiple business groups who may be using different subscriptions. This is beneficial when a single business group might not require the full amount of available bandwidth associated with an ExpressRoute circuit. It also adds greater flexibility to increase the number of IPs required at a later time.

In Figure 2, MSIT-CORP-ERPROVIDER-1 represents the Azure subscription that is designated as the provider subscription for owning ExpressRoute circuits. Within the subscription are Azure resource groups, named for Azure regions, into which are placed the ExpressRoute circuits that are then used by the consumer subscriptions that belong to our business units.

Illustration of Microsoft’s implementation of the provider subscription model for ExpressRoute networks: from left to right; Microsoft on-premises to CPE to Few sub interfaces to MSEE to multiple physical Azure locations.

Figure 2. The Microsoft Digital implementation of the provider subscription model for ExpressRoute networks

The Provider Subscription has a fairly simple setup. For each region where we’re going to connect an ExpressRoute for connectivity, we create a resource group. This resource group owns the resources associated with that virtual network connectivity. We work with our central IP management team to allocate a large IP range for each circuit, from which smaller IP address ranges are assigned. As a particular region grows, we might need an additional circuit added. Table 1 provides an example of our configuration and naming conventions.

Table 1. Naming conventions for Azure subscriptions, resource groups, and ExpressRoute circuits

Subscription name

Resource group name

ExpressRoute circuit name

MSIT-Corp-ERProvider-01

WestUS

MSIT-CORP-WUS-Ckt-1

MSIT-Corp-ERProvider-01

WestUS

MSIT-CORP-WUS-Ckt-2

MSIT-Corp-ERProvider-01

NorthEurope

MSIT-CORP-NEU-Ckt-1

MSIT-Corp-ERProvider-01

NorthEurope

MSIT-CORP-WUS-Ckt-2

MSIT-Corp-ERProvider-01

SouthEastAsia

MSIT-CORP-SEA-Ckt-1

MSIT-Corp-ERProvider-01

SouthEastAsia

MSIT-CORP-SEA-Ckt-2

MSIT-Corp-ERProvider-01

SouthCentralUS

MSIT-CORP-SCUS-Ckt-1

MSIT-Corp-ERProvider-01

SouthCentralUS

MSIT-CORP-SCUS-Ckt-2

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-1

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-2

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-3

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-4

MSIT-Corp-ERProvider-01

EastUS2

MSIT-CORP-EUS2-Ckt-1

MSIT-Corp-ERProvider-01

EastUS2

MSIT-CORP-EUS2-Ckt-2

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-1

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-2

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-3

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-4

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-1

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-2

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-3

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-4

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-5

As you can see from the table, we have ExpressRoute circuits configured across various parts of the United States, as well as Europe and Southeast Asia. We used a defined naming convention for subscriptions, resource groups, and ExpressRoute circuits—this gives us a standardized environment and simplifies ExpressRoute maintenance and usage. All of these resources can be managed within the same centralized shared provider subscription. As you can see, some regions have more circuits configured for them than others, reflecting our ability to use this model to grow based on need. As might be expected, the WestUS2 region has the most circuits (five) because it’s located near the main Microsoft campus in Redmond, Washington.

Getting our internal customers connected

As part of pivoting our Azure ExpressRoute services for our internal customers, we had to provide a way to request an ExpressRoute connection for their subscription. This was initially implemented by creating a website that could gather the pertinent information from them to implement the request and then pass it on for use in the implementation process. Figure 3 shows the New Network form on our internal website that collects the following information about the request:

  • Request Name. Plain text to identify the request, similar to “New Finance IT ExpressRoute Request.”
  • Azure Subscription. The Subscription ID (GUID) for the subscription to connect.
  • Network Type. Drop down to pick between our corporate network and our perimeter network.
  • Region. Choose the region they need for the virtual network connection.
  • Network Size. /26 or a /27 virtual network. Customers that require something larger can contact us to request an exception.
  • Service Name. The service associated as defined within our Azure service hierarchy.
  • Associated Users. We required at least two named contacts for notifications, follow-up, and to contact if there are any concerns regarding the configuration. Our security team will also use these contacts as required for detected compliance issues.
Screen capture of a web form named New Network. The form includes the following fields: Request Name,  Azure Subscription,  Network Type,  Region,  Network Size,  Service Name,  Associated Users. At the bottom there is button labeled Validate user.

Figure 3. The New Network creation page for requesting ExpressRoute services

After we have this information, we use some simple automation that creates a resource group within the internal customer’s subscription to host the new virtual network resources. We create a virtual network of the appropriate size in the consumer subscription and link it with the appropriate ExpressRoute circuit in the provider subscription. This is now a spoke in the hub and spoke model.

The example in Figure 4 shows the ERNETWORK resource group, which is created in the consumer subscription. In this example, a virtual network named MSIT-CORP-WUS-VNET-1 is also created in the ERNETWORK resource group, along with other network resources, such as a public IP address and a gateway. As the circuit owner, we create an authorization key in the provider subscription that can be used by the consumer subscription to access the ExpressRoute circuit. Finally, the authorization key is used to create a connection named MSIT-CORP-WUS-Conn-1 in the consumer subscription to link the gateway in the consumer subscription to the ExpressRoute circuit in the provider subscription.

Illustration of an Azure ExpressRoute network infrastructure - the ERNETWORK resource group,  which contains the following: a virtual network,  a Public Address,  a Gateway,  and a Virtual Network Connection.

Figure 4. An example of ExpressRoute network infrastructure in Azure

Managing ExpressRoute more efficiently

Our goal with the provider subscription model is multilayered, but it ultimately comes down to efficiency in management. By hosting ExpressRoute circuits in a single subscription, we’ve experienced several benefits to reduce the amount of Azure peering overhead, enable faster ExpressRoute circuit provisioning, reduce routing overhead, and optimize our use of finite routable IP space. We’ve also found several benefits:

  • ExpressRoute circuits are hosted in fewer subscriptions, which provides centralized management and reduces sprawl.
  • Dedicated bandwidth can be shared between multiple subscriptions, reducing costs and overhead.
  • Provisioning is simplified. There’s one place to create ExpressRoute circuits, and having fewer individually created ExpressRoute connections reduces the network configuration overhead in modifying network routes and configuring IP subnets.
  • The model can easily be used by vendors, partners, and other third parties without the requirement to exchange account authentication information.
  • Troubleshooting and management is simplified because our central IT team owns and has access to the subscription where the physical ExpressRoute circuit is hosted.
  • We can allocate virtual networks in consumer subscriptions in a round-robin fashion to maintain load balancing throughout the network environment.
  • The relationship between Microsoft Digital and the business groups is more accurately reflected. As the group responsible for network configuration and management, we have primary control over how our network is being used by the business groups.

You might also be interested in

Best practices for implementing Zero Trust at Microsoft
June 03, 2021

Best practices for implementing Zero Trust at Microsoft

Watch video
Microsoft Digital builds a better wide area network with Microsoft Azure
June 02, 2021

Microsoft Digital builds a better wide area network with Microsoft Azure

Read blog
Verifying devices in a Zero Trust model
May 28, 2021

Verifying devices in a Zero Trust model

Read Article
Verifying identity in a Zero Trust model
May 27, 2021

Verifying identity in a Zero Trust model

Read Article