Copy Custom VM Images on Azure

  • February 15, 2018
  • Views

    8,150

Background

We recently worked with Namogoo, a cloud security and analytics startup that works with e-commerce and retail companies to protect against ads that direct customers to competitors. Virtual machines (VMs) with custom images are a key part of Namogoo’s system.  These images include all proprietary software and configurations required for their offerings. The VMs that host these images are used in an elastic manner and auto-scaled as required by traffic patterns.

Namogoo made the decision to migrate a large portion of their infrastructure from AWS to Azure with support from Microsoft. This code story explains how we created an azure-cli extension that simplifies the process of distributing VM images globally.

The Challenge

The idea behind custom images is that you bring up a VM, do all the installation and configuration needed for the solution based on it to work and finally, package the VM’s disks as an image to be re-used later on. Since the image already includes the specific customization for a solution, using it enables the environment to come online faster compared to using other methods (like using a “stock” image followed by init scripts). This tie factor becomes more important in complex systems that use auto-scaling because when more machines are needed to serve traffic, you want them available sooner rather than later.

A custom image is a resource in a specific region; as a result, it needs to be replicated in all target regions.

Replicating images across regions in Azure requires users to:

  1. Create disk snapshots from a source image
  2. Create a storage account in the destination region
  3. Create a container in the destination storage account
  4. Copy a snapshot blob to the destination container
  5. Create a snapshot of the blob in the destination region
  6. Create an image from the snapshot
  7. Clean up temporary resources

Steps 2-6 need to be performed across each destination region to which we are copying the image. Furthermore, these steps are just an outline and the actual procedure is between 10 to 15 steps, depending on the scenario. As you can see, the process is not straightforward. Although there are a few automation scripts available on the web, they require some tinkering to get the job done.

Namogoo updates their images frequently and deploys them in many different regions and this whole process is tedious and time-consuming on Azure as compared to AWS, which was making it difficult to migrate more workloads to Azure. The objective of our solution was to create a way to copy images between regions that would be simple to use.

The Solution

Together with the relevant product groups at Microsoft, we decided to create a new azure-cli command extension that consolidates the process outlined above into a single command.

Example of the image copy command:

A command extension is a relatively new feature in the Azure CLI that offers new functionality not available when the CLI is first installed. This feature is similar to Git subcommands, vscode extensions, etc. that allow users to extend the core application.

The list-available command lists publicly available extensions:

A specific extension can be added to your CLI using the add command:

Developing an extension is very similar to developing other CLI commands and consists of the following components:

  1. Parameter definition
  2. Logic
  3. Help text

You can find more information on Azure CLI extensions on GitHub.

Image Copy Extension

Using Internal Commands

Our extension encapsulates the process of copying the image by using existing CLI commands. To keep the extension independent and decoupled from the implementation of commands used, we decided to invoke commands externally (as if they were called from a script).

The extension works in steps, where each step goes through the following stages:

  1. Prepare the command to run
  2. Run the command
  3. Process the result

For example, the following function creates a new resource group:

Performance

Copying an image from one region to another can take some time since the actual copying is performed asynchronously by Azure storage infrastructure on spare bandwidth between the regions. If the regions are close, this process can be relatively quick and only take around 4 minutes. However, it can take considerably longer between far apart regions (for example, copying between Europe and Australia can sometimes take 30 minutes).

In cases where it is necessary to copy an image to several regions, it makes sense to do so in parallel to save time and avoid redundant actions on the source image.

Using the extension

First, install the extension:

Then, call it as you would any other az command:

The extension piggybacks on the image parent command and extends it to add the copy functionality. The parameters listed describe the source and destination. The last one, --cleanup  deletes all temporary resources created in the process like snapshots, storage accounts, etc. In addition, all resources created (temporary and permanent) are tagged by the created_by=image-copy-extension  tag.

While the command is running you can see the various stages as well as the progress of the parallel copy process to each region. Below is an example of such an output:

Source Disk

The whole process of copying an image relies upon the existence of the disk used to create the source image. Currently, there’s no way to access the actual resources behind the source image so we have to use the disk originally used to create the image.

Conclusion

Copying custom VM images between regions is a basic requirement for many Azure deployments; as such, the learning in this code story is reusable in many other scenarios. The image-copy extension has helped to streamline Namogoo’s DevOps processes on Azure. It will also enable other customers to reduce deployment times by providing a quick and easy way to copy images across regions in Azure.

Adding functionality to azure-cli is simple and extensions can be shared publicly for everyone to use, or privately within your organization. If you try out image-copy-extension, let us know what you think in the comments below.

Resources


Cover image source

Related Articles

Leave a reply

Your email address will not be published. Required fields are marked *

2018-10-11 15:20:23

Tamir Kamara says:

Hi, This issue is related to a change in the core cli module which is about to be fixed. After it will be released (maybe already) just update your cli and it should work fine. For now I think you can try using version 45.

2018-10-04 00:31:53

Pavan Kumaar says:

Hi, I am using the latest version of Azure CLI (2.0.46). While executing the above commands, I have received the error: command ended with an error: ['C:\\Program Files\\Microsoft SDKs\\Azure\\CLI2\\python.exe', '-m', 'azure.cli', 'storage', 'blob', 'copy', 'start', '--source-uri', None, '--destination-blob', 'techsignvm-image_os_disk_snapshot.vhd', '--destination-container', 'snapshots', '--account-name', 'northeuropee85644y16lp77', '--sas-token', 'se=2018-10-04T08%3A24Z&sp=rwlacup&spr=https&sv', '--output', 'json']Any way to fix this?

2018-08-20 11:59:35

Tamir Kamara says:

Hi, A new feature was added recently enabling saving the copied images in a different subscription. Please use the github repo to add requests and feedback.

2018-07-16 03:50:41

Pradeep says:

Thanks, How can we further extend when we are working with multiple subscriptions ? Did you try this scenerio

2018-06-19 04:25:51

Tamir Kamara says:

Hi Muhammad, You're right - currently the extension doesn't include data-disks in the images since we believe it's less common scenario than the case where you have only an OS disk. Please open an issue on the GitHub repo asking for this feature, and we'll continue the discussion there: https://github.com/Azure/azure-cli-extensions

2018-06-18 12:08:36

Muhammad Rashid says:

az image copy command works fine when image have single OS disk. However if image have data disk as well then the 'az image copy' command don't copy the data disk. To move the data disks, lot of effort is required. Is there any method by which 'az image copy' command can move both os and data disks.looking forward for guidance?

2018-06-18 11:02:34

Boris Cornejo says:

Thank you!! I did exactly that and it works!

2018-06-18 10:28:59

Tamir Kamara says:

Thank you for the comment. Glad to see that you find the post valuable. I think the issue mention was fixed in version 0.0.7 released recently. Please try updating the extension installed on your machine with the command: az extension update --name "image-copy-extension". If the problem persist please open an issue on the GitHub project: https://github.com/Azure/azure-cli-extensions

2018-06-18 09:44:30

Boris Cornejo says:

Thank you so much for this. I tried it on regular Azure and it works great, however, when I try it on Government Azure it does not. I am entering the correct location for where I need to transfer the image to, however, it gives me the following error: az image copy --source-resource-group DaaS_PoC --source-object-name DaaS_PoC_Image_CentosDesktop_v2 --target-location usgovt exas --target-resource-group "images-repo-rg" --cleanup Getting os disk id of the source vm/image Creating source snapshot Getting sas url for the source snapshot Creating resource group: image-copy-rg command failed: ['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe', '-m', 'azure.cli', 'group', 'create', '--name', 'image-copy-rg', '--location', 'eastus', '--output', 'json', '--tags', 'created_by=image-copy-extension'] output: ERROR: The provided location 'eastus' is not available for resource group. List of available regions is 'usgoviowa,usgovvirginia,usdodeast,usdodcentral,usgovtexas,usgovarizona'.Command '['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe', '-m', 'azure.cli', 'group', 'create', '--name', 'image-copy-rg', '--location', 'eastus', '--output', 'json', '--tags', 'created_by=image-copy-extension']' returned non-zero exit status 1. Traceback (most recent call last): File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\knack\knack\cli.py", line 197, in invoke File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\azure-cli-core\azure\cli\core\commands\__init__.py", line 336, in execute File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\six\six.py", line 693, in reraise File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\azure-cli-core\azure\cli\core\commands\__init__.py", line 310, in execute File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\azure-cli-core\azure\cli\core\commands\__init__.py", line 171, in __call__ File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\knack\knack\commands.py", line 109, in __call__ File "C:\Users\trdai\AppData\Local\Temp\pip-install-972ovukd\azure-cli-core\azure\cli\core\__init__.py", line 420, in default_command_handler File "C:\Users\boris.o.cornejo.sosa\.azure\cliextensions\image-copy-extension\azext_imagecopy\custom.py", line 68, in imagecopy create_resource_group(transient_resource_group_name, 'eastus') File "C:\Users\boris.o.cornejo.sosa\.azure\cliextensions\image-copy-extension\azext_imagecopy\custom.py", line 150, in create_resource_group run_cli_command(cli_cmd) File "C:\Users\boris.o.cornejo.sosa\.azure\cliextensions\image-copy-extension\azext_imagecopy\cli_utils.py", line 35, in run_cli_command raise ex File "C:\Users\boris.o.cornejo.sosa\.azure\cliextensions\image-copy-extension\azext_imagecopy\cli_utils.py", line 21, in run_cli_command cmd_output = check_output(cmd, stderr=STDOUT, universal_newlines=True) File "subprocess.py", line 336, in check_output File "subprocess.py", line 418, in run subprocess.CalledProcessError: Command '['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe', '-m', 'azure.cli', 'group', 'create', '--name', 'image-copy-rg', '--location', 'eastus', '--output', 'json', '--tags', 'created_by=image-copy-extension']' returned non-zero exit status 1.I have never entered anything on the command about 'eastus', however, it appears to default to it. Has any seen this before?

2018-06-15 03:15:25

Marlon says:

Thank you for the great post

2018-06-09 21:50:08

Tamir Kamara says:

Thank you for your comment Samir. Yes, if you need to update an image you need to bring a new VM online, make all the changes and recapture it.

2018-06-07 17:10:32

Samir H says:

This is awesome, just what I needed. One question that may be unrelated to the post, but what do you do when you need to update your image? Let's say there are so many Windows updates that are not captured in the image, or a piece of software I forgot to add to the image, and adding those after the VM has been deployed off the image takes too long. Do I need to deploy a VM, perform the installs and updates, and then generalize the image again, thus creating a new image? And then delete the existing (old) image and redo the image copy using your extension? Thanks!