September 2015

Volume 30 Number 9

DevOps - Enabling DevOps on the Microsoft Stack

By Michael Learned | September 2015

There’s a lot of buzz around DevOps right now. An organization’s custom software is critical to providing rich experiences and useful data to its business users. Rapidly delivering quality software is no longer an option, it’s a requirement. Gone are the days of lengthy planning sessions and development iterations.  Cloud platforms such as Microsoft Azure have removed traditional bottlenecks and helped commoditize infrastructure. Software reigns in every business as the key differentiator and factor in business outcomes. No organization, developer or IT worker can or should avoid the DevOps movement.

DevOps is defined from numerous points of view, but most often refers to removing both cultural and technology barriers between development and operations teams so software can move into production as efficiently as possible. Once software is running in production you need to ensure you can capture rich usage data and feed that data back into development teams and decision makers.

There are many technologies and tools that can help with DevOps. These tools and processes support rapid release cycles and data collection on production applications. On the Microsoft stack, tools such as Release Management to drive rapid, predictable releases and Application Insights help capture rich app usage data. This article will explore and shed some light on critical tools and techniques used in DevOps, as well as the various aspects of DevOps (as shown in Figure 1).

Figure 1 The Various Aspects of DevOps

The Role of DevOps

Most organizations want to improve their DevOps story in the following areas:

• Automated release pipelines in which you can reliably test and release on much shorter cycles. 
• Once the application is running in production, you need the ability to respond quickly to change requests and defects.
• You must capture telemetry and usage data from running production applications and leverage that for data-driven decision making versus “crystal ball” decision making.

Are there silos in your organization blocking those aspects of DevOps? These silos exist in many forms, such as differing tools, scripting languages, politics and departmental boundaries. They intend to provide separation of duties and to keep security controls and stability in production.

Despite their intentions, these silos can sometimes impede an organization from achieving many DevOps goals, such as speedy, reliable releases and handling and responding to production defects. In many cases, this silo structure generates an alarming amount of waste. Developers and operations workers have traditionally worked on different teams with different goals. Those teams spend cycles fixing issues caused by these barriers and less time focused on driving the business.

Corporate decision makers need to take a fresh look at the various boundaries to evaluate the true ROI or benefits these silos intend to provide. It’s becoming clear the more you can remove those barriers, the easier it will be to implement DevOps solutions and reduce waste.

It’s a challenge to maintain proper security, controls, compliance and so on while balancing agility needs. Enterprise security teams must ensure data is kept secure and private. Security is arguably as important as anything else an organization does. 

However, there’s an associated cost for every security boundary you build. If security boundaries are causing your teams waste and friction, those boundaries deserve a fresh look to ensure they generate ROI. You can be the most secure organization in the world, but if you can’t release software on time you’ll have a competitive disadvantage.

Balancing these priorities isn’t a new challenge, but it’s time for a fresh and honest look at the various processes and silos your organization has built. Teams should all be focused on business value over individual goals.

The Release Pipeline

The release pipeline is where your code is born with version control, then travels through various environments and is eventually released to production. Along the way, you perform automated build and testing. The pipeline should be in a state where moving changes to production is transparent, repeatable, reliable and fast. This will no doubt involve automation. The release pipeline might also include provisioning the application host environment.

Your release pipeline might not be optimized if these factors are present:

• Tool and process mismatches, whereby you have different tools and processes in place per environment. (For example, the dev teams deploy with one tool and ops deploy with another.)
• Manual steps can introduce error, so avoid them.
• Re-building just to deploy to the next environment.
• You lack traceability and have issues understanding which versions have been released.
• Release cycles are lengthy, even for hotfixes.

Provisioning

Provisioning containers is sometimes considered an optional part of a release pipeline. A classic on-premises scenario often exists in which an environment is already running to host a Web application. The IIS Web server or other host and back-end SQL Server have been running through numerous iterations. Rapid releases into these environments deploy only the application code and subsequent SQL schema and data changes needed to move the appropriate update levels. In this case, you’re not provisioning fresh infrastructure (both IIS and SQL) to host the application. You’re using a release pipeline that disregards provisioning and focuses only on the application code itself.

There are other scenarios in which you might want to change various container configuration settings. You might need to tweak some app pool settings in IIS. You could implement that as part of the release pipeline or handle it manually. Then you may opt to track those changes in some type of versioning system with an Infrastructure-as-Code (IaC) strategy.

There are several other scenarios in which you would want to provision as part of an automated release pipeline. For example, early in development cycles, you might wish to tear down and rebuild new SQL databases for each release to fully and automatically test the environment.

Cloud computing platforms such as Azure let you pay only for what you need. Using automated setup and tear down can be cost-effective. By automating provisioning and environmental changes, you can avoid error and control the entire application environment. Scenarios like these make it compelling to include provisioning as part of a holistic release management system.

There are many options and techniques for including provisioning as part of your release pipeline. These will differ based on the types of applications you’re hosting and where you host them. One example is hosting a classic ASP.NET Web application versus an Azure Web app or some other Platform-as-a-Service (PaaS) application such as Azure Cloud Services. The containers for those applications are different and require different tooling techniques to support the provisioning steps.

Infrastructure as Code

One popular provisioning technique is IaC. An application is an executable that can be compiled code, scripts and so on combined with an operational environment. You’ll find this environment yields many benefits. 

Microsoft recently had Forrester Research Inc. conduct a research study on the impact of IaC (see bit.ly/1IiGRk1). The research showed IaC is a critical DevOp component. It also showed provisioning and configuration is a major point of friction for teams delivering software. You’ll need to leverage automation and IaC techniques if you intend to completely fulfill your DevOps goals.

One of the traditional operational challenges is automating the ability to provide appropriate environments in which to execute applications and services, and keeping those environments in known good states. Virtualization and other automation techniques are beneficial, but still have problems keeping nodes in sync and managing configuration drift. Operations and development teams continue to struggle with different toolsets, expertise and processes.

IaC is based on the premise that we should be able to describe, version, execute and test our infrastructure code via an automated release pipeline. For example, you can easily create a Windows virtual machine (VM) configured with IIS using a simple Windows PowerShell script. Operations should be able to use the same ALM tools to script, version and test the infrastructure.

Other benefits include being able to spin up and tear down known versions of your environments. You can avoid troublesome issues because of environmental differences between development and production. You can express the application environment-specific dependencies in code and carry them along in version control. In short, you can eliminate manual processes and ensure you’ve tested reliable automated environment containers for your applications. Development and operations can use common scripting languages and tools and achieve those efficiencies.

The application type and intended host location will dictate the tooling involved for executing your infrastructure code. There are several tools gaining popularity to support these techniques, including Desired State Configuration (DSC), Puppet, Chef and more. Each helps you achieve similar goals based on the scenario at hand. 

The code piece of IaC could be one of several things. It could simply be Windows PowerShell scripts that provision resources. Again, the application types and hosting environment will dictate your choices here.

For Azure, you can use Cloud Deployment Projects that leverage Azure Resource Management APIs to create and manage Azure Resource Groups. This lets you describe your environments with JSON. Azure Resource Goups also let you manage group-related resources together, such as Web sites and SQL databases. With cloud deployment projects, you can store your provisioning requirements in version control and perform Azure provisioning as part of an automated release pipeline. Here are the sections that make up the basic structure of a provisioning template:

{
  "$schema": "https://schema.management.azure.com/schemas2015-01-01/
    deploymentTemplate.json#",
  "contentVersion": "",
  "parameters" { },
  "variables": { },
  "resources": [ ],
  "outputs": { }
}

For more information on templates, go to bit.ly/1RQ3gvg, and for more on cloud deployment projects, check out bit.ly/1flDH3m.

The scripting languages and tooling are only part of the changes needed to successfully adopt an IaC strategy. Development and operations teams must work together to integrate their work streams toward a common set of goals. This can be challenging because historically operations teams have focused on keeping environments stable and development teams are more focused on introducing new features into those environments. Sophisticated technologies are emerging, but the foundation of a successful IaC implementation will depend on the ability of the operations and development teams to effectively collaborate.

Release Orchestration

Release Management is a technology in the Visual Studio ALM stack. It’s really more of a concept whereby you can orchestrate the various objects and tasks that encompass a software release.  A few of these artifacts include the payload or package produced by a build system, the automated testing that happens as part of a release pipeline, approval workflows, notifications and security governance to control environments closer to production.

You can use technologies such as DSC, Windows PowerShell scripts, Azure Resource Manager, Chef, and so on to manage environment state and install software and dependencies into running environments. In terms of tooling provided by Visual Studio ALM, think of Release Management as the service that wraps around whatever technologies and tools you need to execute the deployments. Release Management might leverage simple command-line or Windows PowerShell scripts, use DSC, or even execute your own custom tools. You should aim to use the simplest solution possible to execute your releases.

It’s also a good practice to rely on Windows PowerShell because it’s ubiquitous. For example, you can use Windows PowerShell scripts as part of a release pipeline to deploy Azure Cloud Services. There are a lot of out-of-the-box tools with Release Management (see Figure 2), but you also have the flexibility to create your own.

Figure 2 Tools and Options Available for Release Management

Release Management can help you elegantly create an automated release pipeline and produce reliable automated application releases. You can also opt to include provisioning.  The Release Management tooling with Visual Studio and Team Foundation Server can help you orchestrate these artifacts into the overall release transaction. It also provides rich dashboard-style views into your current and historical states. There’s also rich integration with Team Foundation Server and Visual Studio Online.

Where Does DSC Fit In?

There has been a lot of press about DSC lately. DSC is not, however, some all-encompassing tool that can handle everything. You’ll use DSC as one of the tools in your DevOps structure, not the only tool.

You can use DSC in pull or push modes. Then you can use the “make it so” phase to control the server state. Controlling that state can be as simple as ensuring a file or directory exists, or something more complex such as modifying the registry, stopping or starting services, or running scripts to deploy an application. You can do this repeatedly without error. You can also define your own DSC resources or leverage a large number of built-in resources.

DSC is implemented as a Local Configuration Manager (LCM), running on a target node, accepting a Management Object File (MOF) configuration file and using it to apply configuration to the node itself.  So there’s no hard-coupled tool. You don’t even have to use Windows PowerShell to produce the MOF. 

To start using DSC, simply produce the MOF file. That will eventually describe the various resources to execute, which end up written mostly in Windows PowerShell. One of the big advantages of DSC on Windows Server-based systems is the LCM is native to the OS, giving you the concept of a built-in agent. There are also scenarios for leveraging DSC with Linux. See Figure 3 for an example of separating the configuration data for the DSC script.

Figure 3 Separate Configuration Data Within a DCS Script

Configuration InstallWebSite
{
  Node $AllNodes.NodeName
  {
    WindowsFeature InstallIIS
    {
      Ensure = "Present"
      Name = "Web-Server"
    }
  }
}
InstallWebSite –ConfigurationData .\config.ps1
Where config.ps1 contains
$ConfigData = @{
  AllNodes = @(
  @{
    NodeName = “localhost”
  })
}

DSC can be an important piece of a release pipeline if it has the resources available to help support your deployment. With on-premises or IaaS applications, DSC is an excellent choice to help control the environment configuration and support your deployment scenarios. 

Still DSC isn’t meant to be used for every scenario. To put this in context, if you’re deploying Azure PaaS resources, it’s recommended you use Azure Resource Manager to get the VMs started and the networking configured. This isn’t something DSC is designed for. Once the VMs are running, you can use DSC to get the local configuration the way you want it and ensure the configuration elements you care about don’t change.

 Monitor with Application Insights

Once an application and environment is in production, it’s critical to collect data and monitor the operational health. You also need to understand usage patterns. This data is critical to managing a healthy service. Collecting and monitoring this data is an important piece of DevOps. For example, Microsoft has used production data to improve the Visual Studio Online teams. This rich data helps the Visual Studio Online teams ensure service availability, demonstrates to them how developers are using the service and informs decisions on feature prioritization. You can read more about the Microsoft DevOps journey at bit.ly/1AzDL9V.

Visual Studio Application Insights adds an SDK to your appli­cation and sends telemetry to the Azure Portal. It supports many different platforms and languages, including iOS, Android, ASP.NET and Java. You can capture performance data, application uptime and various usage analytics. You can show this rich data to decision makers and stakeholders to help make better decisions, detect issues and continuously improve your applications. You can read more about Application Insights at bit.ly/1IbRnrF.

Figure 4 and Figure 5 show examples of the types of data collected by Application Insights.

Figure 4 Application Insights Can Provide Data on Users and Page Views

Figure 5 Application Insights Also Monitors Web Tests

Wrapping Up

DevOps helps teams drive toward continuous delivery and leverage data from running applications to help make better-informed decisions. This article has examined various prominent Microsoft technologies you can use to achieve these goals:

• Release Management lets you use any technology to drive deployments. These technologies include simple Windows PowerShell scripts, DSC configurations or even third-party tools such as Puppet.
• Infrastructure-as-Code strategies help development and operations teams efficiently work together.
• Visual Studio Application Insights gives you a mechanism to capture rich data from running applications, to help stakeholders understand application health and examine usage patterns to drive informed decision making.

These technologies can help you greatly improve your DevOps maturity. You’ll also need to blend an appropriate set of technologies while working to overcome cultural barriers.

Additional Resources

• To learn more about Infrastructure as Code, listen to Brian Keller’s discussion on Channel 9 at bit.ly/1IiNqmr.
• To learn more about Azure Resource Group Deployment Projects, check out bit.ly/1flDH3m.
• To learn more about TFS Planning, Disaster Avoidance and Recovery, and TFS on Azure IaaS, check out the guide at vsarplanningguide.codeplex.com.
• To learn more about Config as Code for DevOps and ALM practitioners, check out vsardevops.codeplex.com.

Micheal Learned is a Visual Studio ALM Ranger currently focused on DevOps and Microsoft Azure. He has worked on numerous software projects inside and outside of Microsoft for more than 15 years. He lives in central Illinois and devotes his free time to helping the community, as well as relaxing with his daughter, two sons and wife. Reach him on Twitter at twitter.com/mlhoop.

Thanks to the following technical experts for reviewing this article: Donovan Brown (Microsoft), Wouter de Kort (Independent Developer), Marcus Fernandez (Microsoft), Richard Hundhausen (Accentient), Willy-Peter Schaub (Microsoft) and Giulio Vian (Independent Developer)