With migration to Azure, it’s important that Microsoft IT ensure network connectivity between on-premises and virtual resources across our environment. To do so, we’ve been using Azure ExpressRoute circuits as part of our network topology. For management of ExpressRoute, we’ve moved to a shared circuit model—business groups access some networking components, but we provision and manage ExpressRoute networking services in a simplified and centralized way. As a result, we’ve reduced overhead and optimized bandwidth.

EXPLORE RELATED CONTENT

Microsoft IT manages one of the largest Azure enterprise environments. We’ve been migrating workloads to Azure and creating new workloads in Azure more and more each year. By the end of the fiscal year 2017, we plan to have 90 percent of our computing resources hosted in Azure. As we’ve migrated many virtual machine resources directly into Azure, the transition has necessitated significant connectivity between our on-premises networks and Azure. We have a large number of ExpressRoute circuits connecting our on-premises resources to our Azure environment, and that number continues to grow very quickly. To ensure proper management of a network environment this large, we’ve moved our Azure networking environment to a shared circuit model—this gives uniform access for our internal business groups to ExpressRoute and other Azure networking components, and it also allows us to centrally manage and monitor how we provision Azure networking.

Understanding ExpressRoute usage at Microsoft

ExpressRoute is the default connectivity method we use for connecting Azure to our internal networks. It’s the best solution in Azure for our large scale network environment, compared to alternatives such as point-to-site or site-to-site virtual networks. We use ExpressRoute to connect Azure to datacenters, corporate headquarters, and regional offices across many different business groups. We use ExpressRoute for two primary purposes:

  • To connect internal or partner on-premises resources to an Azure network containing virtual machines—typically to form a contiguous, isolated network space between Azure and an on-premises network.
  • To provide dedicated and measurable bandwidth between on-premises resources and Azure components.

Difficulties with ExpressRoute hosted in internal business group subscriptions

When we first began our Azure services, ExpressRoute connections had been provisioned at the request of our business groups. We perform the ExpressRoute provisioning and assign IP address ranges to accompany the ExpressRoute circuit within the Azure subscription that belongs to the business group or partner organization. Although this approach has the benefit of leaving the ExpressRoute ownership in the hands of the business group that owns the subscription, it does have some management issues:

  • It doesn’t scale well. As more and more business teams began to embrace Azure, it became quickly clear that a one-to-one relationship between ExpressRoute circuits and subscriptions wasn’t going to scale well.
  • A business team might initially request an IP address range with an ExpressRoute circuit based on their current use of resources, and then require more at a later date. Reconfiguring the ExpressRoute circuit to add a larger IP range requires downtime.
  • It requires a central account in each subscription to manage the circuits. Multiple accounts are complex and time-consuming to manage.
  • There’s not a single point of reference for where our ExpressRoute circuits have been provisioned.
  • ExpressRoute circuits must be provisioned individually. Provisioning tasks don’t only include creating the ExpressRoute circuit—it also includes allocating IP addresses, configuring routers, and other network-related tasks. Having to configure routers for every request quickly started to impact the amount of time required for creating connections.

Using the provider subscription model

The provider subscription model is intended to simplify the provisioning and management of ExpressRoute circuits and Virtual Networks that can leverage them. In this model, we use a single shared provider subscription that owns every physical ExpressRoute circuit. These ExpressRoute circuits are then used by other consumer subscriptions that belong to our business groups by connecting Virtual Network to Virtual Network across subscriptions.

A series of objects connected to each other from left to right that illustrate the Microsoft IT ExpressRoute design 1 to many mapping.A series of objects connected to each other from left to right that illustrate the Microsoft IT ExpressRoute design 1 to many mapping.

Figure 1. The ExpressRoute shared circuit model

Implementing a shared provider subscription

To better manage the virtual networks for our ExpressRoute environment, we created shared provider subscriptions to specifically manage the ExpressRoute circuits. These provider subscriptions are owned by our central Microsoft IT Infrastructure team. This team provides services out to the wider business units within Microsoft IT and the larger corporate teams within all of Microsoft.

The process for linking an ExpressRoute circuit to an external Azure consumer subscription is a straightforward process and, after it’s complete, the business group can use the ExpressRoute connectivity in their own Azure environment. When we organize the ExpressRoute circuits this way, we can share connectivity between multiple business groups who may be using different subscriptions. This is beneficial when a single business group might not require the full amount of available bandwidth associated with an ExpressRoute circuit. It also adds greater flexibility to increase the number of IPs required at a later time.

In Figure 2, MSIT-CORP-ERPROVIDER-1 represents the Azure subscription that is designated as the provider subscription for owning ExpressRoute circuits. Within the subscription are Azure resource groups, named for Azure regions, into which are placed the ExpressRoute circuits that are then used by the consumer subscriptions that belong to our business units.

Illustration of Microsoft’s implementation of the provider subscription model for ExpressRoute networks: from left to right; Microsoft on-premises to CPE to Few sub interfaces to MSEE to multiple physical Azure locations.

Figure 2. The Microsoft IT implementation of the provider subscription model for ExpressRoute networks

The Provider Subscription has a fairly simple setup. For each region where we’re going to connect an ExpressRoute for connectivity, we create a resource group. This resource group owns the resources associated with that virtual network connectivity. We work with our central IP management team to allocate a large IP range for each circuit, from which smaller IP address ranges are assigned. As a particular region grows, we might need an additional circuit added. Table 1 provides an example of our configuration and naming conventions.

Table 1. Naming conventions for Azure subscriptions, resource groups, and ExpressRoute circuits

Subscription name

Resource group name

ExpressRoute circuit name

MSIT-Corp-ERProvider-01

WestUS

MSIT-CORP-WUS-Ckt-1

MSIT-Corp-ERProvider-01

WestUS

MSIT-CORP-WUS-Ckt-2

MSIT-Corp-ERProvider-01

NorthEurope

MSIT-CORP-NEU-Ckt-1

MSIT-Corp-ERProvider-01

NorthEurope

MSIT-CORP-WUS-Ckt-2

MSIT-Corp-ERProvider-01

SouthEastAsia

MSIT-CORP-SEA-Ckt-1

MSIT-Corp-ERProvider-01

SouthEastAsia

MSIT-CORP-SEA-Ckt-2

MSIT-Corp-ERProvider-01

SouthCentralUS

MSIT-CORP-SCUS-Ckt-1

MSIT-Corp-ERProvider-01

SouthCentralUS

MSIT-CORP-SCUS-Ckt-2

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-1

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-2

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-3

MSIT-Corp-ERProvider-01

CentralUS

MSIT-CORP-CUS-Ckt-4

MSIT-Corp-ERProvider-01

EastUS2

MSIT-CORP-EUS2-Ckt-1

MSIT-Corp-ERProvider-01

EastUS2

MSIT-CORP-EUS2-Ckt-2

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-1

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-2

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-3

MSIT-Corp-ERProvider-01

WestCentralUS

MSIT-CORP-WCUS-Ckt-4

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-1

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-2

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-3

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-4

MSIT-Corp-ERProvider-01

WestUS2

MSIT-CORP-WUS2-Ckt-5

As you can see from the table, we have ExpressRoute circuits configured across various parts of the United States, as well as Europe and SouthEast Asia. We used a defined naming convention for subscriptions, resource groups, and ExpressRoute circuits—this gives us a standardized environment and simplifies ExpressRoute maintenance and usage. All of these resources can be managed within the same centralized shared provider subscription. As you can see, some regions have more circuits configured for them than others, reflecting our ability to use this model to grow based on need. As might be expected, the WestUS2 region has the most circuits (five) because it’s located near the main Microsoft campus in Redmond, Washington.

Getting our internal customers connected

As part of pivoting our Azure ExpressRoute services for our internal customers, we had to provide a way to request an ExpressRoute connection for their subscription. This was initially implemented by creating a website that could gather the pertinent information from them to implement the request and then pass it on for use in the implementation process. Figure 3 shows the New Network form on our internal website that collects the following information about the request:

  • Request Name. Plain text to identify the request, similar to “New Finance IT ExpressRoute Request.”
  • Azure Subscription. The Subscription ID (GUID) for the subscription to connect.
  • Network Type. Drop down to pick between our corporate network and our perimeter network.
  • Region. Choose the region they need for the virtual network connection.
  • Network Size. /26 or a /27 virtual network. Customers that require something larger can contact us to request an exception.
  • Service Name. The service associated as defined within our Azure service hierarchy.
  • Associated Users. We required at least two named contacts for notifications, follow-up, and to contact if there are any concerns regarding the configuration. Our security team will also use these contacts as required for detected compliance issues.
Screen capture of a web form named New Network. The form includes the following fields: Request Name,  Azure Subscription,  Network Type,  Region,  Network Size,  Service Name,  Associated Users. At the bottom there is button labeled Validate user.

Figure 3. The New Network creation page for requesting ExpressRoute services

After we have this information, we use some simple automation that creates a resource group within the internal customer’s subscription to host the new virtual network resources. We create a virtual network of the appropriate size in the consumer subscription and link it with the appropriate ExpressRoute circuit in the provider subscription.

The example in Figure 4 shows the ERNETWORK resource group, which is created in the consumer subscription. In this example, a virtual network named MSIT-CORP-WUS-VNET-1 is also created in the ERNETWORK resource group, along with other network resources, such as a public IP address and a gateway. As the circuit owner, we create an authorization key in the provider subscription that can be used by the consumer subscription to access the ExpressRoute circuit. Finally, the authorization key is used to create a connection named MSIT-CORP-WUS-Conn-1 in the consumer subscription to link the gateway in the consumer subscription to the ExpressRoute circuit in the provider subscription.

Illustration of an Azure ExpressRoute network infrastructure - the ERNETWORK resource group,  which contains the following: a virtual network,  a Public Address,  a Gateway,  and a Virtual Network Connection.

Figure 4. An example of ExpressRoute network infrastructure in Azure

Managing ExpressRoute more efficiently

Our goal with the provider subscription model is multilayered, but it ultimately comes down to efficiency in management. By hosting ExpressRoute circuits in a single subscription, we’ve experienced several benefits to reduce the amount of Azure peering overhead, enable faster ExpressRoute circuit provisioning, reduce routing overhead, and optimize our use of finite routable IP space. We’ve also found several benefits:

  • ExpressRoute circuits are hosted in one subscription, which provides centralized management and reduces ExpressRoute sprawl.
  • Dedicated bandwidth can be shared between multiple subscriptions, reducing costs and overhead.
  • Provisioning is simplified. There’s one place to create ExpressRoute circuits, and having fewer—hundreds fewer—individually created ExpressRoute connections reduces the network configuration overhead in modifying network routes and configuring IP subnets.
  • The model can easily be used by vendors, partners, and other third parties without the requirement to exchange account authentication information.
  • Troubleshooting and management is simplified because our central IT team owns and has access to the subscription where the physical ExpressRoute circuit is hosted.
  • We can allocate virtual networks in consumer subscriptions in a round-robin fashion to maintain load balancing throughout the network environment.
  • The relationship between Microsoft IT and the business groups is more accurately reflected. As the group responsible for network configuration and management, we have primary control over how our network is being used by the business groups.

For more information

Microsoft IT

http://www.Microsoft.com/ITShowcase

ExpressRoute FAQ

Create and modify an ExpressRoute circuit

About VPN Gateway

 

© 2016 Microsoft Corporation. All rights reserved. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.


You might also be interested in

Strategies for migrating SAP systems to Microsoft Azure
May 06, 2019

Strategies for migrating SAP systems to Microsoft Azure

Read case study
Building an agile and trusted SAP environment on Microsoft Azure
May 06, 2019

Building an agile and trusted SAP environment on Microsoft Azure

Read case study
Activating the super powers of Azure at Microsoft
April 02, 2019

Activating the super powers of Azure at Microsoft

Learn more
Speaking of security: Cloud migration
April 02, 2019

Speaking of security: Cloud migration

Watch webinar