Connect(); 2016

Volume 31 Number 12

[ALM and DevOps]

Secure and Deliver with Rugged DevOps

By Jean-Marc Prieur, Sam Guckenheimer; 2016

Security can be a scary topic. According to the Microsoft Security Intelligence Report (bit.ly/2drid6a), last year 17.9 percent of reporting computers encountered security threats. And the urgency to protect is greater than ever. The time to compromise is almost always days or less, if not minutes or less, finds Verizon in its “2016 Data Breach Investigations Report” (vz.to/2dnpNk8).

Often there’s a perceived conflict between DevOps practices, which aim for speed, and security practices, which emphasize thoroughness. The conflict doesn’t have to exist. In fact, a number of vendors have been working to make DevOps pipelines more secure. And this is part of a movement toward Rugged DevOps. This article discusses four extensions newly available through the Visual Studio Team Services (VSTS) Marketplace.

Let’s start with an overview of a Rugged DevOps pipeline (see Figure 1).

Figure 1 The Rugged DevOps Workflow

The goal of a Rugged DevOps pipeline is to allow development teams to go fast without breaking things by introducing unwanted vulnerabilities. For case studies of teams implementing Rugged DevOps, see chapter 22 of “The DevOps Handbook” (IT Revolution Press, 2016) by Gene Kim, Jez Humble, Patrick Debois and John Willis  (bit.ly/2dUXJW3).

Package Management

Just as a team uses version control as the single source of truth for current source code, it can rely on a package manager as the unique source of binary components. By using binary package management, a development team can create not only a local cache of approved components, but also make this a trusted feed for the continuous integration (CI) pipeline. Package management—now in VSTS and coming to Team Foundation Server (TFS)—is an integral part of the component workflow. The Package Management extension is available from the Visual Studio Marketplace (bit.ly/1VLXIzG).

The Role of OSS Components

Developers today are much more productive than ever, due to the wide availability of reusable open source software (OSS) components. There’s now a practical approach to reuse, with runtimes available on Windows and Linux such as .NET Core and Node.js. At the same time, the productivity of reusing OSS comes with the risk that the reused dependencies bring security vulnerabilities. For example, thesnyk.io service claims that 76 percent of its users find security vulnerabilities in their applications due to the versions of the Node.js packages they consume. 

In the world of OSS, there’s a new concept, sometimes called Software Composition Analysis (SCA), which is depicted in Figure 2. It involves scenarios like this:

As a developer or dev lead, when I consume an OSS component, create a dependency or consume dependencies, I want to: start with the latest correct version in order to avoid any old vulnerabilities or license misuse; validate that they are in fact the correct binaries for this version; in the release pipeline, validate binaries to ensure they’re correct and to keep a traceable bill of materials; and in the event of a vulnerability, be notified immediately and be able to correct and redeploy automatically in order to prevent any security vulnerability or license misuse from reused software.

Figure 2 Safely Creating Open Source Dependencies

Two vendors that now provide services for managing OSS with VSTS and TFS are WhiteSource and Veracode. Both have extensions available in the Visual Studio Marketplace. 

Solution Spotlight: WhiteSource Extension for VSTS and TFS

For a team consuming external packages, the WhiteSource Extension specifically addresses the questions of open source security, quality and license compliance. In a world where most breaches target known vulnerabilities in known components, this is essential hygiene for consuming open source. We’ll discuss some of its capabilities.

Continuously Detects All Open Source Components in Your Software The WhiteSource extension (bit.ly/2dEMCC0) will automatically detect all open source components—including their transitive dependencies—every time you run a build. This means you can generate a comprehensive inventory report within minutes, based on the last build you ran (see Figure 3). It also gives your security, DevOps and legal teams full visibility into your organization’s software development process.

Figure 3 WhiteSource Component Inventory

Alerts on Open Source Security Vulnerabilities and Their Fixes When a new security vulnerability is discovered, WhiteSource automatically generates an alert and provides targeted remediation guidance (see Figure 4). This can include links to patches, fixes, relevant source files and even recommendations to change system configuration to prevent exploitation. 

Figure 4 WhiteSource Detection of Vulnerable Components

Automatically Enforces Open Source Security and License Compliance Policies According to a company’s policies, WhiteSource automatically approves, rejects or triggers a manual approval process every time a new open source component is added to a build. Developers can set up policies based on parameters such as security-vulnerability severity, license type or library age. As soon as a developer attempts to add a problematic open source component, the service will send an alert and fail the build.

For searching online repositories such as GitHub and Maven Central, WhiteSource also offers an innovative browser extension. Even before choosing a new component, a developer can see its security vulnerabilities, quality, and license issues, and whether it fits a company’s policy.

Solution Spotlight: Extension for VSTS: The Fortify Static Code Analyzer (SCA)

HPE Security Fortify SCA provides static analysis for application security testing through VSTS and TFS (bit.ly/2dEWEOW). This makes software security a seamless part of the coding process by empowering developers to find security vulnerabilities earlier in the DevOps lifecycle.

Fortify SCA provides a comprehensive set of software security analyzers that search for violations of security-specific coding rules and guidelines. Development groups and security professionals use it to analyze application source code for security issues (see Figure 5). Fortify SCA identifies root causes of software-security vulnerabilities and delivers accurate, risk-ranked results with line-of-code remediation guidance.

Figure 5 Visual Studio Team Services Build Tasks for HPE Security Fortify SCA

Fortify on Demand also delivers application Security as a Service (SaaS). Fortify on Demand tasks automatically submit static and dynamic scan requests to the application SaaS platform (see Figure 6). For static assessments, the project is uploaded to Fortify on Demand. For dynamic assessments, Fortify on Demand uses the application’s pre-configured URL.

Figure 6 HPE Security Fortify on Demand Vulnerability Detection

Balancing Speed and Depth

In the past, security scanning was typically an “over-the-wall” activity, done perhaps only once per release. This creates a horrible pattern in which security specialists find large batches of issues at exactly the time when developers are under the most pressure to ignore them and release. Rugged DevOps strives to make all quality activity—including security—continuous and automated.

All of the extensions described here can fully scan the team’s source code. There are multiple points to integrate scanning into the team’s workflow.

Pull requests (PRs) are the way DevOps teams submit changes. Prior to the PR, a developer needs to be able to see the effect of code changes and be confident they’ll merge correctly and not introduce new issues. In a DevOps process, each PR is typically small and merges are continual, so the master branch of code can stay fresh. Ideally, the developer can check for security issues prior to the PR. WhiteSource facilitates this for validating dependencies with its binary fingerprinting; Checkmarx provides an incremental scan of changes; and Veracode has the concept of a developer sandbox. These approaches allow a developer to experiment with changes before submitting them.

Similarly, CI needs to be optimized for speed to give the development team immediate feedback of any build breaks. Just as in the PR flow, when the scanning is fast enough, it can and should be integrated into the CI build definition. A failed scan can break the build, and security issues can be fixed right away to restore the build to green.

At the same time, continuous delivery (CD) needs to be thorough. In VSTS, CD is typically managed through either release definitions, which progress the build output across environments, or via additional build definitions that can be scheduled—perhaps daily—rather than triggered with each commit. In either case, the definition can perform a longer static analysis scan (see Figure 7). The full code project can be scanned and any errors or warnings reviewed offline without blocking the CI flow.

Figure 7 The Build Definition Can Trigger a Static Analysis Scan of Source Code

Solution Spotlight: Checkmarx Extension for VSTS

The Checkmarx extension for VSTS (bit.ly/2dVyuDg) allows developers to not only scan all source code, but also to just scan new or modified code. This incremental scanning capability is a key enabler for developers in CI environments as it reduces scan times from hours to only a few minutes.

Once the analysis is complete, Checkmarx delivers a concise report detailing the security posture of the scanned code on the VSTS project summary page, which allows developers to address and mitigate the vulnerabilities (see Figure 8).

Figure 8 Visual Studio Team Services Build Report with Checkmarx Scan Results

Checkmarx also provides developers with the “Best Fix Location” in order to minimize the time to remediate. This includes a visual chart of the data flow graph that indicates the ideal location in the code to address multiple vulnerabilities within the data flow in a single line of code (see Figure 9).

Figure 9 Checkmarx Best Fix Location in Data Flow Graph

With rule customization, development teams also can modify the existing vulnerability detection preset queries to enhance detection and accuracy. Checkmarx describes the rules in a simple C# syntax.

Secure Code Is Only Part of the Picture

Secure code is necessary, but not sufficient to achieve Rugged DevOps. The Verizon “2016 Data Breach Intelligence Report” makes clear the many attack vectors that skilled criminals use to compromise their targets. In addition to protecting your code, it’s essential to protect credentials and secrets. In particular, phishing is becoming ever more sophisticated.

There are several operational practices that a team ought to apply to protect itself, as we’ll discuss now.

Authentication and Authorization Use multi-factor authentication even across internal domains and just-in-time administration, such as the PowerShell Just Enough Administration (JEA), to protect against escalation of privilege (aka.ms/jea). Different passwords for different user accounts will limit the damage if one set of credentials is stolen.

Use the CI/CD Release Pipeline Do this to rebuild infrastructure so the release pipeline and cadence can also contain damage. If you manage Infrastructure as Code with Azure Resource Manager or use Azure PaaS or a similar service, then your pipeline will automatically create new instances and destroy them, giving attackers no place to hide inside your infrastructure (bit.ly/2dEY5wR). VSTS will encrypt the secrets in your pipeline, and you should rotate the passwords just as you would other credentials.

Manage Permissions Do this to secure the pipeline with role-based access control just as you would for your source code. You want to control who can edit the build and release definitions you use for production.

Dynamic Scanning This is the process of testing the running application with known attack patterns. For example, OWASP Zed (bit.ly/1fjloVy) is a popular open source dynamic scanner to check against the primary vulnerabilities that owasp.orgtracks. Two of the partners discussed in this article, HPE Security and Veracode, provide dynamic scanning services, as well.

Monitoring Production This is a key DevOps practice. The specialized services for detecting anomalies related to intrusion are known as Security Information and Event Management. Azure Security Center focuses on the security incidents related to the Azure cloud (bit.ly/2dzcj5r).

Solution Spotlight: Veracode Extension for VSTS

The Veracode Application Security Platform is a SaaS that enables developers to automatically scan an application for security vulnerabilities. Veracode provides static application security testing (SAST), dynamic application security testing (DAST) and SCA, allowing development teams to assess both first-party code and third-party components for security risk (see Figure 10).

Figure 10 Visual Studio Team Services Build Report with Veracode Scan Results

The Veracode VSTS extension (bit.ly/2dme4Vr) allows teams to configure continuous and automated assessment of their applications as a build or release step from their CI/CD pipeline. The build or release step can also be configured to automatically stop a build or release based on application security policy, allowing developers to integrate security testing into a fully automated CD pipeline. In addition, Veracode offers “Developer Sandbox” scans to provide results on an individual developer’s changes before submission. 

Why Integrate Security Scanning into Your DevOps Pipeline?

Because it’s now practical. With the new extensions in the VSTS Marketplace, you can make security scanning a continuous part of your team’s release pipeline. Unlike in the past, when scans were infrequent and generated a wall of issues right before a release, you can address warnings and errors as they occur. By addressing the security warnings in small batches—either with each PR or even daily—you can reduce the work in process and address component and code security continuously.

In order to optimize for speed, DevOps promotes the idea of shift-left. In other words, whatever is worth doing, is worth doing continuously as part of a DevOps workflow. When you make changes, make them and version them with code. This creates one source of truth: your source repo and trusted package management store. Whenever you update, update the single source of truth.

According to the 2016 “State of DevOps Report” (bit.ly/28NI32i) from Puppet, high performers, in addition to higher agility and reliability outcomes, also had better security outcomes. By integrating information security (InfoSec) objectives into the daily work of Dev and Ops, they spent 50 percent less time remediating security issues. If you integrate security into your DevOps pipeline, then you have an automatic way to create, test and deploy updates. This will shorten your time to remediate when new vulnerabilities are discovered. Security updates become just like other updates and can follow the same automated flow.

The practices make reuse safe. When you scan packages continuously, you can depend on them and know that you aren’t picking up their vulnerabilities. Moreover, when new vulnerabilities are discovered in the wild, you can be notified immediately so security intelligence becomes actionable. You can pick up the new package versions, modify your code as needed, test and release—without needing to wait for attackers to discover your vulnerability.

DevOps started by breaking down silos between development and operations. Now you can break down the next wall, between DevOps and InfoSec. Rather than build up security debt that needs to be addressed too late, you can prevent it from creeping into the pipeline.

The combination of these benefits helps you go fast. By integrating security automation into your pipeline, you can use it as an accelerator. After all, the bad guys will go for easy targets. And as the old joke has it, if you’re in a camping party in the woods and a bear appears, it’s not the bear you need to outrun, but the bear’s other potential prey.

Sam Guckenheimer works in Visual Studio Cloud Services. He is the author of three books on agile development practices and an e-book on the Visual Studio Team Services journey to cloud cadence. Reach him at samgu@microsoft.com or on Twitter: @samguckenheimer.

Jean-Marc Prieur is a senior program manager at Microsoft envisioning and driving the delivery of experiences in Visual Studio and Visual Studio Team Services focused on controlling technical debt, including architecture analysis tools. Reach him at jmprieur@microsoft.com or on Twitter: @jm_prieur.

Thanks to the following technical experts for reviewing this article: Amit Ashbel, Michael Right, Joanna Rosenberg and Maya Rotenberg
Amit Ashbel, Director of Product Marketing at Checkmarx, Michael Right Senior Product Manager - Fortify at HPE Security, Joanna Rosenberg, Solution Marketing at Veracode and Maya Rotenberg, Head of Marketing at WhiteSource.

Discuss this article in the MSDN Magazine forum