• Article

August 2019

Volume 34 Number 8

[Azure DevOps]

Introducing Azure Deployment Manager

By David Tepper

How complex are your services? Do you have tens of resources split over a bunch of resource groups and regions? Hundreds? Thousands? Does your engineering system make it easy to declaratively define the topology of those resources, how they work together, and how they can be deployed and updated in a safe, scalable fashion?

Currently in preview, Azure Deployment Manager (ADM) provides a new set of features for Azure Resource Manager (bit.ly/2Ib6WHM) and greatly expands your deployment capabilities. If you need to deploy a complex service to several regions, or gain more control over when resources are deployed in relation to one another, or limit your customer’s exposure to bad updates by catching them while in progress, then ADM is for you.

ADM allows you to perform staged rollouts of resources, meaning they’re deployed region by region in an ordered fashion. As part of this rollout, the health of your services is monitored, and if unacceptable performance degradation is detected, deployment automatically stops, allowing you to troubleshoot and reduce the scale of the impact. These tools are used internally by hundreds of Microsoft services to ensure safe, reliable deployments, ensuring high availability and preventing or dramatically reducing service unavailability caused by regressions in updates.

Azure Deployment Manager Concepts

ADM introduces two new concepts to the Azure Resource Manager (ARM) deployment story: service topologies and rollouts.

As depicted in Figure 1, a service topology describes the hierarchy of the resources you deploy to Azure by organizing them into groups. Template and parameter files form the basis of any ARM deployment. These represent the Service Units, or individually deployable chunks of a given service, such as the front end of a service. A group of Service Units comprises a Service, and a Service Topology is nothing more than a group of Services.

Service Topology Structure
Figure 1 Service Topology Structure

ADM makes it easy to define these kinds of complex topologies, even when the Services span multiple regions, subscriptions, and resource groups. Once defined, they can be deployed and then updated any number of times through the use of Rollouts, which defines the structure and timing for how to deploy the Service Units in a staged, safe fashion.

As depicted in Figure 2, a Rollout is composed of Step Groups. Step Groups define the order of deployment and support both sequential and parallel operations. For example, I can have Step Group 2 execute after Step Group 1 (as shown), or have Step Groups 1 and 2 deploy in parallel before deploying Step Group 3.

Rollout Structure
Figure 2 Rollout Structure

As the name implies, Step Groups are further broken down into Steps, which address steps to take before a deployment and steps to take after a deployment. Examples of such steps include:

  • Wait: Pauses deployment for a specified amount of time before continuing, allowing for manual tests, as well as time for resources to bake.
  • Health checks: ADM integrates with existing service monitoring providers to check the health of services as deployments are underway. If the health of a service degrades, deployment is halted.

More steps, including ones that allow for execution of custom code, are coming soon. To better understand these concepts, we’ll work through an example.

Deploy Your First Rollout

Before we get started, it’s a good idea to familiarize yourself with some of the resources and tools you’ll be working with. Here are a few links to help get your environment set up and give you the background you need to work with ARM templates and follow through with the example:

Once you’ve downloaded and installed the example files, you can find the ArtifactStore folder, which contains a templates folder and a binaries folder.

The templates folder contains the ARM template artifacts. 1.0.0.0 and 1.0.0.1 represent the two versions of the binary artifacts. Within each version, there’s a folder for each service (Service East US and Service West US). Each service has a pair of template and parameter files for creating a storage account, and another pair for creating a Web application. The Web application template calls a compressed package, which contains the Web application files. The compressed file is a binary artifact stored in the binaries folder.

The binaries folder contains the binary artifacts with 1.0.0.0 and 1.0.0.1, again representing the two versions of the binary artifacts. Within each version there’s one .zip file for creating the Web application in the West US location, and another .zip file to create the Web application in the East US location.

Even though both the template artifacts and the binary artifacts have two versions, only the binary artifacts are different between the two versions. In practice, binary artifacts are updated more frequently than template artifacts. For a full description of what the contents of these artifact files are, check out the ADM tutorial at aka.ms/admtutorial.

Template artifacts are used by the service topology template, and binary artifacts are used by the rollout template. Both the topology template and the rollout template define an artifact source Azure resource, which is a resource used to point Resource Manager to the template and binary artifacts that are used in the deployment. To simplify this example, one storage account is used to store both the template artifacts and the binary artifacts. Both artifact sources point to the same storage account, so let’s get that set up.

First, create an Azure storage account. For this you can follow the instructions published in the “Quickstart: Upload, Download, and List Blobs Using the Azure Portal” article at bit.ly/2F6NMRt.

Next, create a blob container in the storage account, then copy the two folders (binaries and templates) and the content of the two folders to the blob container. Then get the SAS location of the container using the following steps. From Azure Storage Explorer, navigate to the blob container and right-click the blob container from the left pane, and then select Get Shared Access Signature. Configure Start time and Expiry time and then select Create.

Make a copy of the URL. This URL is the SAS URI needed to populate a field in the two parameter files—the topology parameters file and rollout parameters file—during a later step.

Later in this example, you’ll deploy a rollout. A user-assigned managed identity is needed to perform the deployment actions. This identity must be granted access to the Azure subscription to which you’re deploying the service, and must have sufficient permission to complete the artifact deployment. Let’s create a user-assigned managed identity and configure the access control for your subscription.

Sign in to the Azure portal (portal.azure.com) and create a user-­assigned managed identity. For more on this, see bit.ly/2F7lqqx. From the portal, select Subscriptions from the left menu, and then select your subscription. Select Access control (IAM), and then select Add role assignment. You should see the dialog box in Figure 3.

Creating a User Assigned Managed Identity
Figure 3 Creating a User Assigned Managed Identity

Let’s enter values into the displayed fields. The Role dropdown provides permission to complete the artifact deployment (the Web applications and the storage accounts). For this example, select Contributor from the list. In practice, you want to restrict the permissions to the minimum.

Next, click the Assigned access dropdown and select User Assigned Managed Identity. Then select the user-assigned managed identity you created earlier in the example. Finally, select Save and be sure to write down the Name of your managed identity, as you’ll use this in a later step.

Prepare the Templates

Now that we’ve set up the necessary Azure account resources, it’s time to make some small edits to the provided files so they can correctly target your subscription and storage location. Start by opening \ADMTemplates\CreateADMServiceTopology.Parameters in Visual Studio Code or any text editor. Figure 4 shows the parameters in the Service Topology. Let’s walk through each parameter now.

Service Topology Parameters
Figure 4 Service Topology Parameters

namePrefix: This parameter is used to create the names for the Deployment Manager resources. For example, using the “jdoe” prefix, the service topology name is jdoeServiceTopology. The resource names are defined in the variables section of this template. For this one, use a string with four to five characters. This prefix is used to create unique Azure resource names.

azureResourceLocation: To simplify the example, all resources will share this location unless otherwise specified. Don’t touch this parameter for now.

artifactSourceSASLocation: This field should be populated with the saved SAS URI to the blob container where your service unit template and parameters files are stored for deployment.

templateArtifactRoot: This parameter provides the offset path from the Blob container where the templates and parameters are stored. The default value is templates/1.0.0.0. Don’t change this value unless you want to change the folder structure used when uploading the files to Azure Storage. Relative paths are used in this example, and the full path is constructed automatically by ADM. If, in practice, you want to use absolute paths, ADM supports that, too.

targetSubscriptionID: This is simply the subscription ID to which the Deployment Manager resources are going to be deployed and billed. Use your subscription ID in this example.

Now, open \ADMTemplates\CreateADMRollout.Parameters in Visual Studio Code or any text editor. Enter the parameter values as just described. It’s important that these fields have the same values in both files. There’s a new field in this file that must also be filled in called managedIdentityID. Enter the user-assigned managed identity, replacing the <ManagedIdentity> portion with the saved Name of your managed identity.

First Deployment

It’s time to deploy the two Web sites! Use the Azure CloudShell PowerShell terminal to execute the scripts that follow. Let’s start by running a script to deploy the service topology, like so:

$resourceGroupName = “<Enter a Resource Group Name>”
$location = “Central US” 
$filePath = “<Enter the File Path to the Downloaded Example Files>”
# Create a resource group
New-AzResourceGroup -Name $resourceGroupName -Location “$location”
# Create the service topology
New-AzResourceGroupDeployment `
  -ResourceGroupName $resourceGroupName `
  -TemplateFile “$filePath\ADMTemplates\CreateADMServiceTopology.json” `
  -TemplateParameterFile “$filePath\ADMTemplates\CreateADMServiceTopology.Parameters.json”

Note that New-AzResourceGroupDeployment is an asynchro­nous call. The success message only means the deployment has successfully begun. In a moment I’ll go over how to check the deployment status. For now, verify that the service topology and the underlying resources have been created successfully using the Azure portal, as depicted in Figure 5.

Viewing Hidden Deployment Resources
Figure 5 Viewing Hidden Deployment Resources

Now, let’s deploy the rollout template, as follows:

# Create the rollout
New-AzResourceGroupDeployment `
  -ResourceGroupName $resourceGroupName `
  -TemplateFile “$filePath\ADMTemplates\CreateADMRollout.json” `
  -TemplateParameterFile “$filePath\ADMTemplates\CreateADMRollout.Parameters.json”

We can check the rollout progress using the following script:

# Get the rollout status
$rolloutname = “<Enter the Rollout Name>”
# “adm0925Rollout” is the rollout name used in this example
Get-AzDeploymentManagerRollout `
  -ResourceGroupName $resourceGroupName `
  -Name $rolloutName `
  -Verbose

After the rollout is deployed successfully, you’ll see two more resource groups created, one for each service. Open the Azure portal and browse to the newly created Web applications under the new resource groups created by the rollout deployment. Open the Web application in a Web browser and the site’s HTML will verify the location and the version.

To try out deploying a new version (1.0.0.1) of the Web application, open CreateADMRollout.Parameters.json, then update binaryArtifactRoot to binaries/1.0.0.1. Redeploy the rollout as instructed before. You don’t need to redeploy the Service Topology. Verify the update by navigating to the Web site and noting the version number displayed by the page.

Congratulations, you just deployed two services into multiple regions using a staged rollout and updated them to a new version! Next, let’s walk through integrating health checks into these steps so that if a service update is unhealthy, the deployment will auto­matically stop before impacting your most important regions.

Deploy a Health-Integrated Rollout

Health monitoring providers offer several mechanisms to monitor and alert you to service health issues. Azure Monitor (azure.microsoft.com/services/monitor) is an example of one such service, which fires alerts when certain thresholds are exceeded. Some thresholds are indicative of a problem with service health when exceeded. For example, if you’re deploying a new update to your service and your memory and CPU utilization spike beyond expected levels, Azure Monitor and health monitoring providers like it will notify you of the issues so you can take corrective action.

These health providers typically offer REST APIs so the status of your service’s monitors can be examined programmatically. The REST APIs will either come back with a simple healthy/unhealthy signal (usually determined by the HTTP response code) or with detailed information about the signals it receives.

The new healthCheck step in Azure Deployment Manager allows you to declare HTTP codes that indicate a healthy service, or, for more complex REST results, you can even specify regular expressions that indicate a healthy response if they match. To make this even easier to use, the Azure Deployment Manager team is working closely with the top health monitoring providers to pre-author these HTTP codes and regular expressions, so you can simply copy and paste this part of the healthCheck step if you’re using one of these providers.

The process of getting set up with Azure Deployment Manager health checks is straightforward. First, you create your health monitors through a health service provider of your choice. Then you create one or more healthCheck steps as part of your Azure Deployment Manager rollout and fill out those steps with the following:

  • The URI to the REST API for your health monitors, as defined by your health service provider.
  • Authentication information (only API-key style auth is currently supported).
  • HTTP status codes or regular expressions that define a healthy response.

Finally, you invoke the healthCheck steps at the appropriate time in your Azure Deployment Manager rollout, and ADM takes care of the rest.

Phases of a Health Check

At this point, ADM knows how to query for the health of your service and at what phases in your rollout to do so. However, it also allows for deep configuration of the timing of these checks. A healthCheck step is executed in a simple three-phase process with configurable durations, rich enough to support the health monitoring needs of the largest Microsoft services composing Azure itself.

After a deployment operation is completed, it may not make sense to check for service health right away, as the update needs to reach a steady state. It takes time for services to start emitting health signals to be aggregated by the health monitoring provider into something useful. Likewise, VMs may be starting for the first time, rebooting, or re-configuring based on new data. The result: Service health may oscillate between healthy and unhealthy states. During the Wait phase service health isn’t monitored, so deployed resources have time to bake before beginning the health check process.

It’s often impossible to know how long after startup resources take to become stable, so the Elastic phase provides a flexible period for them to settle into a healthy, steady state. When the Elastic phase begins, Azure Deployment Manager polls the provided REST endpoint for service health periodically. The interval is set to three minutes in the Azure Deployment Manager preview, but will be configurable in the future. If the health monitor comes back with signals indicating that a service is unhealthy, those signals are ignored, the Elastic phase is maintained, and polling continues. Once the health monitor comes back with signals indicating that a service is healthy, the Elastic phase ends and the HealthyState phase begins. Thus, the duration specified for the Elastic phase is the maximum time that can be spent polling for service health before a healthy response is considered mandatory.

During the HealthyState phase, service health is continually polled at the same interval as the Elastic phase. The service is expected to maintain healthy signals from the health-monitoring provider for the entire specified duration. If at any point an unhealthy response is detected, ADM stops the entire rollout and returns the REST response carrying the unhealthy service signals. Once the HealthyState phase duration has ended, the healthCheck is complete, and deployment continues to the next step.

Getting Started

In production, you typically use one or more monitoring providers. To ease health integration, Microsoft has been working with top service health monitoring companies to provide a simple copy/paste solution to integrate health checks into deployments. You can view a list of these health monitoring providers at bit.ly/2XOpR0G.

For this example, you’ll create an Azure Function (bit.ly/2wPgq5d) to simulate a health monitoring service. This function simply takes a status code and returns the same code.

To deploy the Azure Function, use the same Azure CloudShell PowerShell terminal that you used in the previous steps, and then execute the following script. Note that projectName should be less than 12 characters, with only lowercase letters and numbers. It will be reused throughout this example, so be sure to save it. Here’s the script:

$projectName = <Enter a project name that is used to generate Azure resource names>
$location = <Enter the location (i.e. centralus)>
$resourceGroupName = “${projectName}rg”
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateUri “https://armtutorials.blob.core.windows.net/admtutorial/deploy_hc_azure_function.json” -projectName $projectName

To verify and test the Azure function, open the Azure portal and open the resource group. The default name is the project name with “rg” appended. Select the app service from the resource group. The default name of the app service is the project name with “webapp” appended. Expand Functions, and then select HttpTrigger1.

As shown in Figure 6, select </> Get function URL and then click Copy to copy the URL to the clipboard. The URL should look similar to:

 

https://myhc0417webapp.azurewebsites.net/api/healthStatus/{healthStatus}?code=hc4Y1wY4AqsskAkVw6WLAN1A4E6aB0h3MbQ3YJRF3XtXgHvooaG0aw==

Working with Azure Functions
Figure 6 Working with Azure Functions

Now replace {healthStatus} in the URL with a status code. In this example, use unhealthy to test the unhealthy scenario, and use either healthy or warning to test the healthy scenario. Create two URLs, one with the unhealthy status, and the other with the healthy status. For example:

https://myhc0417webapp.azurewebsites.net/api/healthStatus/unhealthy?code=hc4Y1wY4AqsskAkVw6WLAN1A4E6aB0h3MbQ3YJRF3XtXgHvooaG0aw==

https://myhc0417webapp.azurewebsites.net/api/healthStatus/healthy?code=hc4Y1wY4AqsskAkVw6WLAN1A4E6aB0h3MbQ3YJRF3XtXgHvooaG0aw==

Save both, as you’ll need them to complete this example. You can test the URLs by simply opening them in a browser. You should see a simple page that reads “Status:” followed by the value you put into the URL.

Perform the Second Deployment

To speed things along, the PowerShell scripts that follow will reference an updated service topology and a new rollout file that contains a health check step. This step occurs after the West US deployment, but before the East US deployment. Thus, if the health check fails, the East US service will never be deployed, preventing users in that region from experiencing the failure. This health check is configured to look for “Status: healthy” as a health response from the Azure Function, and anything else as an unhealthy response.

You’ll pass the URLs to your Azure Function as parameters when kicking off the script. If you want to download these files to inspect them, you can check out the Service Topology template (bit.ly/2KdzW45) and the Rollout template (bit.ly/31wbP6e).

First, deploy the updated Service Topology. Because the script is rather long, feel free to copy it from here: aka.ms/admhealthdeploy. Here’s the script:

$projectName = <Enter the same project name used earlier in this example>
$location = <Enter the location (i.e. centralus)>
$resourceGroupName = “${projectName}rg”
$artifactLocation = “https://aka.ms/admtutorialartifactstore” | ConvertTo-SecureString -AsPlainText -Force
New-AzResourceGroupDeployment `
  -ResourceGroupName $resourceGroupName `
  -TemplateUri “https://armtutorials.blob.core.windows.net/admtutorial/ADMTemplatesHC/CreateADMServiceTopology.json” `
  -namePrefix $projectName -azureResourceLocation $location `
  -artifactSourceSASLocation $artifactLocation

Then verify that the service topology and the underlying resources have been created successfully using the Azure portal, as was shown earlier in Figure 5.

Next, we’ll use the unhealthy Azure Function URL created earlier, along with the Name of the User Assigned Managed identity, to deploy the rollout. As before, feel free to copy this script from aka.ms/admhealthdeploy. Figure 7 shows the script.

Figure 7 Deploying a Health Integrated Rollout

$projectName = <Enter the same project name used earlier in this example>
$location = <Enter the location (i.e. centralus)>
$managedIdentityID = <Enter the user-assigned managed identity Name>
$healthCheckUrl = <Enter the health check Azure function URL>
$healthCheckAuthAPIKey = $healthCheckUrl.Substring($healthCheckUrl.IndexOf(“?code=”)+6, $healthCheckUrl.Length-$healthCheckUrl.IndexOf(“?code=”)-6)
$healthCheckUrl = $healthCheckUrl.Substring(0, $healthCheckUrl.IndexOf(“?”))
$resourceGroupName = “${projectName}rg”
$artifactLocation = “https://aka.ms/admtutorialartifactstore” | ConvertTo-SecureString -AsPlainText -Force
# Create the rollout
New-AzResourceGroupDeployment `
  -ResourceGroupName $resourceGroupName `
  -TemplateUri “https://armtutorials.blob.core.windows.net/admtutorial/ADMTemplatesHC/CreateADMRollout.json” `
  -namePrefix $projectName `
  -azureResourceLocation $location `
  -artifactSourceSASLocation $artifactLocation `
  -managedIdentityID $managedIdentityID `
  -healthCheckUrl $healthCheckUrl `
  -healthCheckAuthAPIKey $healthCheckAuthAPIKey
As with the previous rollout, you can check the progress using this script:
$projectName = <Enter the same project name used earlier in this example>
$resourceGroupName = “${projectName}rg”
$rolloutName = “${projectName}Rollout”
# Get the rollout status
Get-AzDeploymentManagerRollout `
  -ResourceGroupName $resourceGroupName `
  -Name $rolloutName `
  -Verbose

After enough time has passed, checking the status of the rollout will show a failure. This is expected because the unhealthy Azure Function was used! You’ll also see that one resource group was created for the West US service, but you won’t see the East US resource group because the rollout was automatically stopped before continuing on to other regions. Repeat this deployment with the healthy status URL, and after it has completed, you’ll see that both services have successfully deployed!

That’s a Wrap

ADM allows you to perform complex, multi-region, multi-subscription deployments that integrate with your existing service health monitoring solutions, using Azure’s built-in ARM templates. No additional tools are required, and it’s completely free to use. Get started by reviewing the documentation at aka.ms/admdocs, or explore the full tutorial at aka.ms/admtutorial.

David Tepper is a principal program manager at Microsoft on the OneDeploy team in Azure. He has expertise in machine learning and has worked on both shell and deployment technologies for Windows.

Thanks to the following Microsoft technical experts for reviewing this article: Cristina del Amo Casado, Jonathan Gao, Devesh Guha Oleti Muni, Sriram Ramaswamy
Cristina del Amo Casado is principal program manager in Azure Compute at Microsoft.
Jonathan Gao (jgao@microsoft.com) is senior technical writer for Azure Resource Manager at Microsoft.
Devesh Guha Oleti Muni (Devesh.Oleti@microsoft.com) is a senior software engineer with the Azure team at Microsoft.

Discuss this article in the MSDN Magazine forum