Connect(); Special Issue 2018

Volume 33 Number 13

[Azure]

Deploy Your Code The Right Way with Azure Pipelines

By Micheal Learned and Andy Lewis; Special Issue 2018

Modern applications are increasingly complex systems that involve multiple technology stacks and cloud-native services. Orchestrating an automated release pipeline for these systems can be challenging. Azure Pipelines provides powerful, easy-to-use continuous integration (CI) and continuous delivery (CD) services you can use to build and test your app and then deploy to your intended targets. In this article, we’ll provide an overview of the key concepts of Azure Pipelines and discuss deployment scenarios for various Azure services. We’ll also walk through a detailed scenario for creating a pipeline for a .NET Core app that targets Docker containers in Azure Kubernetes Service (AKS). Finally, we’ll show you the next generation of CI/CD in which your YAML pipeline is configured as part of your code.

Azure Pipelines provides flexibility for configuring your build and release workflows. You can choose from multiple version control providers such as GitHub, Azure Repos, Bitbucket and Subversion. Most languages and application frameworks are supported, including .NET, Java, JavaScript, Python, PHP, Ruby, Xcode, C++ and Go. You can create pipelines that deploy anywhere, including any cloud provider or on-premises deployment target. You can use any package management repository such as NuGet, npm, and Maven to produce and consume packages. Azure Pipelines allows you to configure and customize your CI/CD solution to fit virtually any scenario. Built-in pipeline tasks, third-party tasks, custom tasks, and file-based or inline PowerShell and Bash scripting all help you build workflows for your pipelines. Templates provide you with an easy way to quickly group and reuse a set of tasks.

Azure Pipelines Key Concepts

In Azure Pipelines you define build (CI) and release (CD) workflows for your apps. These workflows provide a framework for orchestrating and automating your release pipelines. A typical scenario is for your CI build pipeline to pull code from your version control repository, compile the code, run tests and other analysis processes, and finally publish artifacts, such as files, binaries, and executables. A CD release pipeline is automatically triggered and uses the published artifacts as the deployment payload for testing and production environments. Azure Pipelines allows you to create simple or complex CI/CD pipelines to deploy your apps to a wide range of deployment targets, including a variety of Azure services such as Web servers and databases. You define various stages of a release pipeline such as Dev, QA and Production. It’s important to note that build and release pipelines are flexible, and you can define pretty much any workflow you want with them.

Agents are Windows, Linux, and macOS machines (or containers) that run the jobs in your build and release pipelines. You can install and self-host your agents on physical or virtual machines (VMs) either on-premises or in the cloud (for example, Azure). Or to save yourself time, you can use Microsoft-hosted Linux, macOS and Windows agents. For open source projects, we provide up to 10 parallel Microsoft-hosted jobs for free (see the links at the end of this article for details).

Build and release tasks are the basic building blocks of your pipelines. They’re packaged scripts that do the work in your pipeline. Azure Pipelines comes with many common tasks for performing actions to build, test, package and deploy code. You can deeply customize your process with tasks that support file-based or inline Bash, PowerShell, and Python scripts. You can even run your Windows batch file scripts. You can use tasks provided by Azure Pipelines, leverage third-party tasks from Visual Studio Marketplace and, if necessary, create your own custom tasks.

Build and release templates provide reusable patterns to help you quickly configure CI/CD for common scenarios such as building an Android app or deploying an Azure Web app. A template is essentially a pre-defined collection of tasks with parameters preset to meet the needs of common app types and targets. You can customize the templates and even put your custom workflows into reusable templates to support the specific needs of your organization.

Service connections provide a secure mechanism for connecting external services to Azure Pipeline tasks. These external services can include your Azure subscriptions, Jenkins or other CI systems, and remote services such as file servers, to name just a few. A typical scenario is to create an Azure Resource Manager (ARM) service connection to enable Azure Pipelines to deploy your apps to various Azure service targets, such as Web Apps, Azure Virtual Machines, Azure SQL Database or any other resource in your Azure subscriptions.

Azure Deployment Targets

Azure provides a variety of services that serve as deployment targets for your apps. App Service, Windows or Linux virtual machines, containers, and many other Azure services are valid deployment targets. Docker containers can be used with App Service, VMs and with Kubernetes via AKS.

Virtual machines (Linux and Windows) are common deployment targets for a wide range of apps. Azure Pipelines supports VMs as deployment targets in Azure, on-premises or even on other cloud providers. You install and configure a deployment agent on your VMs, and then you can hook up Azure Pipelines deployments to your VMs. Agents and Azure Pipelines communicate over asymmetric encryption and HTTPS to provide a secure deployment approach. Deployment groups allow you to organize the servers that host your app. Each node in the deployment group hosts an Azure Pipelines agent, and you can deploy to them. One example for the use of deployment agents is to deploy Web app code to a group of Web servers. You can also install agents on physical machines and use Azure Pipelines to deploy to them with similar patterns.

App Service is an Azure service that allows you to host applications in the language of your choice without having to manage infrastructure, and you can run APIs, mobile back ends, Web Apps and serverless code. Web App for containers allows you to run containerized applications on Windows and Linux. There are many ways to deploy apps to App Service, including with Azure Pipelines. You can use existing Azure Pipelines templates to quickly build a CD pipeline for your app. The App Service Deploy task provides a variety of standard features such as deploying to specific App Service slots, overriding config values at deploy time and executing customs scripts.

AKS provides an Azure-based Kubernetes management service that allows you to deploy and manage containerized applications easily. AKS removes the administration overhead for managing complex container orchestration and simplifies container management by handling health monitoring and maintenance for you. Azure manages Kubernetes masters, and you manage and pay only for the agents. AKS integrates with Azure Container Registry to provide a fully managed container solution.

Azure services and infrastructure can be deployed with templates to provide infrastructure as code (IaC), which yields many benefits such as leveraging version control, CI/CD and automating the deployment of the infrastructure for your apps. ARM templates provide a declarative JSON approach for deploying Azure infrastructure. You can define your networks, security groups, load balancers, AKS and many other Azure infrastructure components with ARM templates.

Azure Pipelines provides support for deploying ARM templates with the Azure Resource Group Deployment task. The task allows you to create pipelines that leverage ARM templates to deploy Azure services to your subscriptions. The task provides several useful configuration options such as the ability to validate templates to prevent errors, deploy in incremental mode to avoid impacting existing Azure resources, and handle creating or updating existing resource groups.

SQL Server is a database deployment target that comes in two forms in Azure. SQL Server can be self-hosted, and Azure SQL Database is an Azure service for Database as a Service (DBaaS) that allows you to use SQL Server without managing infrastructure. Azure Pipelines can deploy to both forms of SQL Server. SQL Server provides several options for deploying schema changes with tasks such as support for dacpac, executing scripts with the sqlcmd utility and using third-party services with Redgate tooling.

Example: Deploy a .NET Core App to an AKS Container Target

.NET Core is a cross-platform (Windows, Linux and macOS) development platform for building device, cloud and Internet of Things (IoT) applications. .NET core is maintained and supported by Microsoft, RedHat and the .NET community. AKS is a managed Kubernetes service for Azure. Kubernetes simplifies the use of containers. You can create a CI/CD pipeline with Azure Pipelines targeting AKS with Docker containers. ARM templates are used to represent the Azure infrastructure, such as AKS and a container registry. Docker images are used to containerize an ASP.NET Web App. This is a moderately complex pipeline, so to help you get started quickly, you’ll find a detailed walk-through that automates the creation of the CI/CD pipeline for this scenario at bit.ly/2T9Y0pT.

An ASP.NET Core Web app is stored in Azure Repos in this example. The ARM templates represent the AKS cluster and the container registry. Using IaC this way allows you to deploy the Azure infrastructure alongside the application, therefore you’re able to manage the Azure Resources and the application in a single CI/CD pipeline. Figure 1 is a view of the basic structures in Azure Repos.

Figure 1 ASP.NET Core App and ARM Templates

The Build pipeline for this scenario contains several steps. You can name the tasks with descriptive labels. Figure 2 shows a set of tasks for setting up a containerized ASP.NET Core app that targets an AKS cluster.

Figure 2 Build Pipeline with Docker and AKS

Get Sources is the first step in the build process. The ASP.NET Core App and the ARM templates are fetched from Azure Repos and brought to the build agent. Cloud-hosted agents keep you from having to manage infra­structure for executing pipelines.

Creating an Azure Container Registry is the next step in the build process. The registry stores and manages Docker images. The next few steps build and publish the Docker image, which in this case contains an ASP.NET Core application. This build step uses an Azure Resource Group Deployment task to deploy an ARM template, which represents the container registry. This task is an example of the flexibility of Azure Pipelines as you’re able to execute a deployment task to set up the registry as part of the build process. The ARM deployment task is set to “incremental mode” by default, so if the container registry already exists, it won’t get created again on subsequent build runs.

Docker tasks are used to build and publish Docker images to the container registry. This step containerizes the ASP.NET Core app. Using an immutable infrastructure with a portable container for the app enables deployment of the container to a wide variety of targets on Windows, most Linux distros and macOS. In this scenario, the container is targeting AKS in Azure.

Helm Charts provide the packaging format for Kubernetes that make it easy to define, version and install Kubernetes applications. Helm, a Kubernetes package manager, is installed on the build agent with the Helm tool installer task. The package and deploy Helm Charts task archives the chart on this step so that the Helm Charts end up published as build artifacts.

Copy ARM templates, which is a Copy task, copies the templates to a staging directory on the build agent. ARM templates are JSON files that represent Azure resources. The ARM template here represents the Kubernetes cluster resource. The ARM templates are stored in Azure Repos, but you can optionally use another version control provider, such as GitHub, for your pipelines.

Publish Build Artifacts is a task that copies the contents of the staging directory to Azure Pipelines. You can store the artifacts on a file share somewhere, but Azure Pipelines are convenient and require no additional set up. The artifacts, in this case, are the ARM templates and the Kubernetes package produced by Helm Charts.

After the build pipeline executes, the release pipeline consumes the artifacts from your build pipeline and deploys the solution. Figure 3shows the various steps in the release pipeline.

Figure 3 Release Pipeline for .NET Core App and Kubernetes

The Azure Resource Group Deployment task is the first step in the release pipeline. This task consumes the AKS ARM template and is the artifact used to deploy the AKS cluster. The ARM template artifacts are available for the release pipeline here because the templates were published as artifacts as part of the build pipeline run.

PowerShell tasks can be used in any build and release pipeline. The PowerShell task here uses the setvariable logging command to initialize some persistent variables so that key bits of common data can be accessed by the subsequent Helm tasks. In this case the script accesses deployment outputs that were created by the previous ARM deployment task. Specifically, variables are set for later use for application routing and Application Insights:

$deploymentOutputs=(ConvertFrom-Json '$(deploymentOutputs)')
$applicationRoutingZone=$deploymentOutputs.applicationRoutingZone.value
$aiKey=$deploymentOutputs.aiKey.value
Write-Host "##vso[task.setvariable
  variable=applicationRoutingZone;]$applicationRoutingZone"
Write-Host "##vso[task.setvariable variable=aiKey;]$aiKey"

Helm tasks orchestrate the final three steps of the release process. The Helm tool installer ensures the agent installs the latest version of the Kubernetes package manager, which includes prerequisites such as Kubectl, the command-line tool for Kubernetes. Helm init and Helm upgrade commands are executed to deploy the application to Kubernetes. Once the release pipeline executes, the ASP.NET Core application is deployed to a Kubernetes cluster, and the application is accessible via URL.

As an option to quickly get started, Azure DevOps Projects can help you automate the creation of a CI/CD pipeline like the one described earlier, especially if you’re not familiar with deploy­ing to Azure services. You use a wizard-driven experience from the Azure portal to choose from a variety of application frameworks and deployment targets. In a few simple steps, you can create a fully functional CI/CD pipeline that builds and deploys your code to various Azure services, such as App Service, VMs, AKS and SQL Database. DevOps Projects allow you to bring your own code or use any of several provided sample applications. You can use DevOps Projects to create a pipeline in Azure Pipelines, and then use that pipeline as a reference architecture for how to do CI/CD for specific application frameworks and Azure service deployment targets. You can also further customize the pipelines. Azure DevOps Projects also creates additional Azure resources such as Application Insights resources for monitoring and an Azure Portal-based dashboard. Azure DevOps Projects documentation can be found at bit.ly/2B7XnpJ.

Config as Code in a YAML Pipeline

In the previous example, you saw how you can use the Azure Portal to generate an end-to-end build pipeline that creates and then feeds artifacts into a release pipeline. The types of pipelines the Azure Portal generates are called “designer pipelines”; these are the drag-and-drop pipelines. This year we introduced a new kind of pipeline you can define in your source code: a YAML pipeline.

The most obvious advantages of a YAML pipeline come from the fact that the workflow lives in your code. You can branch, diff and merge changes to your business logic. If you think that’s cool, wait until the next time someone changes your pipeline and causes a build break or an unexpected outcome. As painful as this scenario can be, you’ll find relief in the fact that you can spot, track and fix the problem just like any other bug in your code!

If your repo contains an azure-pipelines.yml file at the root level, then when you go to create the pipeline, the system picks up the file. You can fork one of our sample app repos to see this in action at aka.ms/get-started-yaml-pipeline. Figure 4 shows a basic ASP.NET Core YAML pipeline. If your GitHub repo doesn’t already have an azure-pipelines.yml file in it, then when you create a new pipeline, the system analyzes the code in your repo and then suggests the kind of pipeline you need, as shown in Figure 5.

Figure 4 A Basic ASP.NET Core YAML Pipeline in a Sample App Repo

Figure 5 Azure Pipelines Looks at Your Code and Suggests a Template for Your YAML Pipeline

To help you migrate from designer pipelines and learn how to adapt them to YAML, tools and information are provided. When you edit your designer pipeline, select a task, and then select View YAML, as shown in Figure 6.

Figure 6 After You Select a Task in the Designer You Can View the YAML

In some cases, you can use this snippet directly. Sometimes (for example if your task uses process parameters, usually generated by a template), you’ll need to manually adjust the YAML.

There’s also a VS Code extension in preview at aka.ms/azure-pipelines-vscode. It’s powered by a language server in case you want to build pre-commit hook tools. Find out more at aka.ms/azure-pipelines-­language-server.

To learn more about what you can do with YAML, see aka.ms/azure-pipelines-yaml-schema and aka.ms/azure-pipelines-task-reference.

Scripts are a very common way to handle CI and CD workflows in code. YAML pipelines are designed to make it as easy as possible for you to integrate scripts wherever you like, in as portable a way as possible. As you get up to speed in YAML, you’ll probably notice both task-centric and script-centric approaches. The decision about which approach to use really comes down to your personal preference.

For simple tasks, you can use a generic cross-platform script:

steps:
- script: echo This is pipeline $(System.DefinitionID)

You can explicitly run a PowerShell or Bash Script. Bash scripts have the advantage of being runnable on multiple platforms. See Figure 7for an example. To learn more, see aka.ms/cross-platform-scripts.

Figure 7 YAML Pipeline with a Bash Script That Can Run on Linux, macOS and Windows Agents

trigger:
  batch: true
  branches:
    include:
    - master
steps:
- bash: |
   echo "Hello world from $AGENT_NAME running on $AGENT_OS"
   case $BUILD_REASON in
    "Manual") echo "$BUILD_REQUESTEDFOR manually queued the build." ;;
    "IndividualCI") echo "This is a CI build for $BUILD_REQUESTEDFOR." ;;
    "BatchedCI") echo "This is a batched CI build for $BUILD_REQUESTEDFOR." ;;
    *) $BUILD_REASON ;;
    esac
  displayName: Hello world

If your team doesn’t need the stages and approval options provided by a designer release pipeline, then you can build, test and deploy all within a single YAML pipeline, as shown in Figure 8.

Figure 8 Build, Test and Deploy from a YAML Pipeline That Lives in Your Codebase

# Build, test and deploy in a YAML pipeline
pool:
  vmImage: 'ubuntu-16.04'
  # replace the hosted Ubuntu pool above with the Windows pool below if
  # you want to deploy to a Windows web app
  # vmImage: 'vs2017-win2016'
variables:
  buildConfiguration: 'Release'
trigger:
- master
steps:
- script: |
    dotnet build --configuration $(buildConfiguration)
    dotnet test dotnetcore-tests --configuration $(buildConfiguration) --logger trx
- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: '**/*.trx'
- task: DotNetCoreCLI@2
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration)
      --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True
- task: AzureRmWebAppDeployment@3
  inputs:
    azureSubscription: 'YourSubscription'
    WebAppName: 'your-webapp'
    Package: $(Build.ArtifactStagingDirectory)/**/*.zip
    # Uncomment the attribute below if you are deploying to a Windows Web app
    # TakeAppOfflineFlag: true

This example works for Linux and Windows Web apps. To adapt it for Windows, just add and remove the relevant comments to change the Microsoft-hosted pool you’re using, and to set the TakeAppOfflineFlag parameter to true to take the app offline before deployment to avoid a file-in-use error on your Web app .DLL file.

Service connections (formerly “endpoints”) can be a tricky part of the process to set up. The following parameters are how you specify the target for your Web app:

azureSubscription: 'YourSubscription'

and

WebAppName: 'your-webapp'

These parameters refer to fields in an Azure Resource Manager service connection. See aka.ms/azure-pipeline-service-connection.

Wrapping Up

Getting started with Azure Pipelines is free. Azure Pipelines provides various free usage tiers for a certain number of monthly minutes and parallel jobs. Additional jobs and minutes can be purchased for a monthly fee. Open source projects receive multiple free parallel jobs. You can find more information about Azure Pipelines pricing at aka.ms/azure-pipelines-pricing.

Deploying your apps to Azure with Azure Pipelines allows you to create reusable CI/CD patterns and reliably automate each step of your deployments. You can create pipelines in multiple ways, use simple or complex workflows, and deploy securely to a wide range of deployment targets. Azure DevOps Projects provide automation for Azure Pipelines to help you quickly get started with deploying your apps to Azure services. YAML pipelines allow you to create configuration-as-code patterns that yield many benefits. To learn more, see aka.ms/sign-up-for-azure-pipelines, aka.ms/azure-devops-project and aka.ms/learn-azure-pipelines.

Micheal Learned has spent more than 20 years working on software engineering projects inside and outside of Microsoft. Learned spends his time focused on family, friends, and all things DevOps and Cloud. You can reach him via Twitter: @mlhoop.

Andy Lewis writes about various things Azure DevOps, including version control and CI/CD pipelines. At Microsoft he’s served customers of Windows, Office and Developer Division. Lewis has designed and developed content, multimedia, and apps for a wide range of audiences at IBM, Borland, Intuit, SAS Institute, and Microsoft.

Thanks to the following Microsoft technical experts for reviewing this article: Jason Conner, Matt Cooper
Jason Conner is a DevOps engineer working out of central Illinois focusing on .NET development, Big Data, and Azure.

Matt Cooper is a program manager based in Raleigh, North Carolina, working on the Azure Pipelines platform, agents, and YAML parser.

Discuss this article in the MSDN Magazine forum