Microsoft Digital Employee Experience (MDEE) protects our high-value corporate assets—beyond just the network. We use shielded virtual machines (shielded VMs) and Host Guardian Services (HGS) in Windows Server 2019 to isolate our data. This ensures that control and administration of infrastructure and environment remain completely isolated from control and administration of data and applications.

Critical data and high risk environments

At MDEE, we classify approximately one percent of the services and data that we host as High Value Assets (HVAs). An HVA is a single isolated environment that provides a secure space for company workloads. Access to HVA data by unauthorized users could negatively affect Microsoft business in a significant way.

In our organization, we host several HVAs for different business groups that need a highly secure environment to prevent unauthorized access or data leaks. Most data in an HVA is classified as highly confidential. HVAs also host data that’s regulated by government policy or other legal restrictions, or that’s physically isolated from other datacenter assets and from our corporate network. A typical HVA can be broken down into several components:

• HVA fabric is the hosting environment for all HVAs. The HVA fabric encompasses the secure hosts, storage, computing, and network services used by all our internal HVA customers.
• HVA stamp is a single instance of an HVA that’s hosted on the HVA fabric. HVA stamps are also called HVA instances.
• Tier 0 holds highly privileged Active Directory resources (such as computer and user accounts) that can give an attacker significant access to the network. These resources include domain controllers, which host the Active Directory database, and highly privileged accounts or groups, such as domain admins or enterprise admins.
• Tier 1 hosts privileged services and systems used by HVAs. These resources provide control over enterprise servers and applications. Tier 1 contains a significant amount of business value hosted in the assets.
• Tier 2 is also called the customer tier. It hosts services that are provided and managed by the internal customer who uses the HVA. Each HVA hosts a unique set of customers and services. Tier 2 services may be privileged in comparison to other systems, but within the scope of the specific HVA, they are the least-privileged services.

HVA topology

A standard HVA host includes the three-tier administrative model and uses the HVA fabric for storage, network, and related services. The components of an HVA are distributed and managed in highly secured datacenters. Each access tier gives a layer of protection against credential theft.

The HVA system is multi-tenant. Each HVA stamp is an isolated environment that’s built for a specific customer or isolated workload. We use isolation techniques to help create clear boundaries between HVA stamps. HVA stamps can be of mixed size (with a different number of virtual machines, different sizes of virtual machines, and so on) and can host a variety of environments. One HVA stamp might host a single Tier 2 service, and others might host full end-to-end environments that have hundreds of servers.

The following figure shows a high-level view of an HVA environment with several HVA stamps.

Using shielded VMs for HVA

To create the private cloud environment that hosts our HVA resources, we use Windows Server 2019, System Center Virtual Machine Manager, and Windows Azure Pack (WAP). Windows Server 2019 introduces the shielded VM feature in Hyper-V. It protects virtual machines from threats outside and inside the fabric. It does this by encrypting disk and virtual machine states so that only virtual machine admins or tenant admins can access them.

By using System Center Virtual Machine Manager and Hyper-V host clusters in our private cloud environment, we can quickly and efficiently provision HVAs. We don’t have to worry about provisioning specific hardware to host HVA resources. The Windows Azure Pack offers a familiar, browser-based interface that our internal customers can use to provision resources. When needed, we provision shielded VMs and provide the computing resources to host an HVA workload.

Guarded fabric health attestation and key release

Shielded VMs are part of the guarded fabric system in Windows Server 2019 Hyper-V. The guarded fabric consists of several layered components:

• Code and boot integrity uses virtualization-based security to allow only approved code to run on the Hyper-V host from the moment it starts.
• Virtualization-based security uses hardware security technology in Windows Server 2019 to create an area that’s isolated from the Windows kernel and other applications to prevent external attacks.
• Trusted Platform Module (TPM) 2.0 is used to securely measure a Hyper-V host’s boot process and code integrity policy. These are then sent to the HGS as part of the health attestation process.
• Host Guardian Service (HGS) acts as an arbitration point for the guarded fabric that contains shielded VMs. HGS provides health attestation for the Hyper-V hosts and key protection for the material that’s required to run the shielded VMs.

Guarded host attestation

As illustrated in the figure below, HGS handles the attestation process for the guarded Hyper-V hosts on which the shielded VMs reside, including key requests and health information. This process ensures the health of the host, the protection of the shielded VM, and the appropriate access for users.

The attestation process includes the following steps:

1. The guarded Hyper-V host sends a key request to the HGS.
2. The HGS replies that it can’t verify that the Hyper-V host is a legitimate host.
3. The Hyper-V host sends its endorsement key to HGS from its TPM module to establish identity, along with health baseline and code integrity policy.
4. If HGS recognizes the identity of the Hyper-V host and considers the baseline and code integrity policy healthy, it supplies a certificate of health to the Hyper-V host.
5. The Hyper-V host re-sends the key request and health certificate to the HGS.
6. The HGS sends an encrypted response back to the Hyper-V host’s virtualization-based security, and the response can be decrypted only by the host hardware security module, to start the shielded VM.

Implementing HVA fabric using shielded VMs

The implementation of HVAs using shielded VMs starts at the datacenter. All HVA servers should be in physically isolated and secure environments. Physical access to the datacenter requires two-person access, and it’s limited to the HVA fabric team and the administrative team.

Physical access implementation

Best practices for implementing physical security components for the HVA include:

• Physical access to the hosting fabric hardware and datacenter floor should require two-person biometric access controls and smart card access to all server cages and racks.
• Physical access to the hosting fabric hardware and datacenter floor by an HVA team admin should require datacenter access tool tickets and a fabric admin escort.
• Datacenter floor access should be granted to only permanent employees.
• Cameras should be used to record all physical access to the datacenter floor and racks.
• The datacenter should have around-the-clock security guards on site—they monitor the facility, datacenter floor, and all access paths.

Hardware implementation

We use only specifically configured hardware in our HVA fabric. Our host hardware runs Windows Server 2019 and Hyper-V. The following table lists the components and management responsibilities.

Hardware components and management responsibilities

Component	Managed by	Description
Fabric host servers with TPM 2.0	Fabric admin team	Host servers are grouped into isolated racks, or pods, and they’re managed by System Center Virtual Machine Manager. They belong to a separate fabric Active Directory Domain Services domain.
Storage systems	Fabric admin team	These are grouped into the same pods as the server infrastructure. HVA fabric storage is provided by System Center Virtual Machine Manager.
Network hardware	Network infrastructure services team in conjunction with the fabric admin team	Firewalls are configured between each layer of the HVA fabric.
Hardware security modules	Network infrastructure services team in conjunction with the fabric admin team	These modules control access to each grouping of Hyper-V host servers that we call a pod. The hardware security modules host secured private keys that participate in the certificate services implementation in HGS. Any administrative function on a hardware security module requires a two-out-of-three security officer quorum.

The following graphic shows the HVA hosting fabric, including HGS, for two primary sites.

The architecture groups together pods of Hyper-V servers as pods, managed by System Center Virtual Machine Manager and fabric domain controllers. The pods are controlled by a group of HGS servers, with access controlled by hardware security modules. We can use this layer separation to separate the administrators of the underlying virtualization fabric from the administrators of the applications and the administrators of the HGS.

Moving forward with HVA using shielded VMs and HGS

We’re experiencing several significant achievements in our HVA environment by using shielded VMs and HGS:

• Dedicated hosting environment (HVA fabric) is a separate host environment for HVAs. It includes specialized configuration and staff who manage the day-to-day operations of all fabric systems. This hosting environment uses the latest and greatest security configurations and systems to help protect HVA customers and workloads.
• HVA stamps isolation is a dedicated environment that holds all necessary services to support a designated workload. This isolated HVA stamp is built to protect itself from all outside threats.
• Role separation keeps every tier of access throughout the fabric, HVA stamps, and related systems isolated. It provides role separation between admins of different functions or types. Fabric admins, network admins, and HVA stamp admins all have isolated credentials and access rights. Most systems also require at least two-person access to change the HVA hosting fabric.
• Privilege elevation mitigation requires different credentials and remote access systems for each tier. IPsec, Group Policy, and silos prevent tier access cross-pollination.
• Network isolation keeps each HVA stamp isolated. All inter-network connections are terminated through a physical firewall. Intra-network connections are protected by IPsec, Windows Firewall, and Group Policy configuration.

• Microsoft extends Azure management to the private cloud with Azure Arc
• Migrating System Center Configuration Manager on-premises infrastructure to Microsoft Azure
• Step by Step – Configuring the Host Guardian Service in Windows Server 2016 – Microsoft Tech Community

Tags: Microsoft Azure, virtual machines