March 2010

Volume 25 Number 03

Security Briefs - Add a Security Bug Bar to Microsoft Team Foundation Server 2010

By Bryan Sullivan | March 2010

One of the most contentious tasks a software development team faces during the course of its products’ lifecycles is triaging bugs. Deciding the relative level of importance of any given bug—and consequently determining the chance that that bug might not be fixed at all in time for release—is a serious matter to everyone involved in the product’s development.

Programmers, testers, architects and program managers all have different viewpoints and base their individual triage decisions on disparate factors such as:

• How much code would have to be regression-tested once the fix is made.
• How close to release the project is.
• How many users would be affected by the change.
• Whether the bug is blocking other issues from being tested or fixed.

I will admit that these are all important factors to consider when triaging functional bugs in product features. However, none of these factors should play any role in determining whether to fix security bugs—that is, bugs that could potentially lead to security vulnerabilities in the product. Classification of security bugs must be objective and consistent. It doesn’t make any difference to an attacker that you found a vulnerability only a week before your code-complete milestone; he’ll exploit it just the same.

This column describes the objective security bug classification system—the “bug bar”—used by Microsoft internal product and online services teams, which is required by the Security Development Lifecycle (SDL). It also shows how you can incorporate this classification system into your own development environment using Microsoft Team Foundation Server 2010.

DREAD

Before I discuss the bug bar as it exists inside Microsoft today, it’s worth describing an earlier Microsoft initiative to classify security bugs: DREAD. DREAD is a mnemonic that stands for:

• Damage Potential
• Reproducibility
• Exploitability
• Affected Users
• Discoverability

Anyone filing a new security bug would assign each of the DREAD parameters a value from 1 to 10, with 10 being the most severe and 1 the least. The values were then averaged to form an overall DREAD rating. For example, say a developer called Doug discovers a blind SQL injection vulnerability in the administration portal page for his team’s new Web application. Doug might classify the vulnerability as shown in Figure 1.

Figure 1 A Developer’s Security Vulnerability Classification

The classification shown in Figure 1 seems fairly straightforward and effective. But consider that Doug’s tester colleague, Tina, might see the exact same vulnerability in a completely different way, as shown in Figure 2.

Figure 2 The Same Bug as Classified by a Tester

Because Tina is aware that there are tools to automate blind SQL injection attacks, she rated Exploitability as 10, whereas Doug saw it as a difficult manual attack and rated the Exploitability as 2. Doug gave Affected Users a 1,because it would only affect a very small portion of the system’s users, but Tina rated it as 10 because the users affected would have administrative rights. Perhaps the parameter of most concern is Damage Potential: both Doug and Tina gave the exact same rationale but scored it with different values!

The question we need to ask at this point is not, “Which team member’s DREAD rating is better?” but rather, “How can we rely on a system that produces such subjective, variable results?” If Tina had been the first person to find the bug, it certainly would have been fixed before release; but if Doug found it first, there’s a good chance it would have been deferred and the application released with the vulnerability. The SDL team concluded that we can’t rely on such a system, and consequently Microsoft developed a more consistent approach: the security bug bar.

The Microsoft Security Bug Bar

The Microsoft security bug bar classifies vulnerabilities based on their effects. The person filing the bug starts by assigning the bug a security effect value from a list of STRIDE values. STRIDE is another mnemonic, in this case used to categorize threats. Unlike DREAD, STRIDE is still very much used by the SDL and is a core component of several SDL tools, including the SDL Threat Modeling Tool. The STRIDE values are:

• Spoofing
• Tampering
• Repudiation
• Information Disclosure
• Denial of Service (DoS)
• Elevation of Privilege (EoP)

However, the broad STRIDE threat category alone isn’t sufficient to classify and triage a bug. In most cases, we need to know whether the bug affects client-side code or server-side code. For example, a DoS attack that takes out a single targeted user would not be considered as severe as one that takes out an entire server. We also need to know some specific scope information for the bug, depending on the STRIDE and client/server classification.

Continuing the DoS vulnerability example, we need to know who can execute the attack (for example, what privilege level) and how long the effects will last. A vulnerability exploitable by an anonymous user is worse than one exploitable only by an authenticated user, and a vulnerability that locks up the affected server until someone physically reboots it is worse than one that just makes it unavailable for a few seconds.

Armed with the primary STRIDE classification plus the additional scope characteristics, the person triaging the bug can now use this information to look up the bug’s severity on the bug bar. Figure 3 shows a sample security bug bar, as published in the Security Development Lifecycle Process Guidance document version 4.1a. The bug bar defines four levels of severity: Critical, Important, Moderate and Low.

Figure 3 Sample Security Bug Bar

Note that in order for this system to work, the bug-tracking database you’re using must have a field for STRIDE security effect. This makes it easy to distinguish security bugs from functional bugs and also to determine the correct bug bar classification. Tracking your bugs’ security effects is so important that there is actually a separate SDL requirement to do this work. Luckily, this is fairly easy in most cases. If you’re using Team Foundation Server (TFS), the next section will show how to add security effect fields and bug bar ratings to Bug work items in your Team Projects.

Adding Bug Bar Functionality to Team Foundation Server

In order to add a bug bar to a TFS Team Project, you need to make a change to the underlying process template that the project was created from. For the purposes of this article, we will assume the project was created from the MSF-Agile for Software Development version 5.0 (beta) template that ships with the TFS 2010 beta 2. However, if you primarily use a different process template, such as the MSF for CMMI Process Improvement template, or any custom third-party template, the techniques used to edit these templates would be the same.

Start by opening the Process Template Manager for the TFS 2010 server you want to work on. You can do this by bringing up the context menu for the server in the Team Explorer window, then navigating to Team Project Collection Settings | Process Template Manager, as shown in Figure 4.

Figure 4 The Process Template Manager for TFS 2010 Serv

In the Process Template Manager, choose the MSF for Agile Software Development v5.0 and click the Download button to bring the template source files down locally. Save them to a folder of your choice.

Once the download is complete, close the Process Template Manager. We want to add the bug bar to the Bug work item type, so open the folder to which you just downloaded the MSF-Agile template, then open the file \WorkItem Tracking\TypeDefinitions\Bug.xml in the XML editor of your choice.

The first task is to add fields for Security Effect, Security Effect Scope and Bug Bar Severity. (Bug Bar Severity should remain distinct from the inherent Severity field for reasons I’ll explain later.) Under the witd:WITD/WORKITEMTYPE/FIELDS element, add the block of XML shown in Figure 5.

Figure 5 Adding Fields for Security Effect, Security Effect Scope and Bug Bar Severity

<FIELD
reportable="dimension"
type="String"
name="Effect"
refname="MSDN.SDL.Security.Effect">
<HELPTEXT>The effect of the security bug</HELPTEXT>
</FIELD>
<FIELD
reportable="dimension"
type="String"
name="EffectScope"
refname="MSDN.SDL.Security.Effect.Scope">
<HELPTEXT>The scope of the effect of the security bug. This value is used to determine the default bug bar severity</HELPTEXT>
</FIELD>
<FIELD
reportable="dimension"
type="String"
name="BugBarSeverity"
refname="MSDN.SDL.Security.Severity.BugBar">
<HELPTEXT>The suggested severity of the bug as determined by the security bug bar</HELPTEXT>
</FIELD>

This code defines the three new fields, including their names, types and help text; but there’s still no logic implemented. Start adding logic by adding allowed-value constraints that lock the possible values for the field.

For Effect, the allowed values are each of the STRIDE values: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service and Elevation of Privilege. It’s also a good idea to add Attack Surface Reduction as an allowed value for cases that may not be vulnerabilities in and of themselves, but are good opportunities to reduce potential future vulnerabilities. Finally, add Not a Security Bug as an allowed value for cases where the bug is just a normal functional bug with no security implications (see Figure 6).

Figure 6 Adding Allowed Values for Effect

<FIELD
reportable="dimension"
type="String"
name="Effect"
refname="MSDN.SDL.Security.Effect">
<HELPTEXT>The effect of the security bug</HELPTEXT>
<ALLOWEDVALUES>
<LISTITEM value="Not a Security Bug" />
<LISTITEM value="Spoofing" />
<LISTITEM value="Tampering" />
<LISTITEM value="Repudiation" />
<LISTITEM value="Information Disclosure" />
<LISTITEM value="Denial of Service" />
<LISTITEM value="Elevation of Privilege" />
<LISTITEM value="Attack Surface Reduction" />
</ALLOWEDVALUES>
<DEFAULT from="value" value="Not a Security Bug" />
</FIELD>

For EffectScope, summarize each of the possible scope items in the bug bar and add these summaries as allowed values for the EffectScope field (see Figure 7).

Figure 7 Adding Allowed Values for EffectScope

<FIELD
reportable="dimension"
type="String"
name="EffectScope"
refname="MSDN.SDL.Security.Effect.Scope">
<HELPTEXT>The scope of the effect of the security bug. This value is used to determine the default bug bar severity</HELPTEXT>
<ALLOWEDVALUES>
<LISTITEM value="Not Applicable" />
<LISTITEM value="Client - Spoofed trusted UI in common/default scenario" />
<LISTITEM value="Client - Spoofed trusted UI in specific other scenario" />
<LISTITEM value="Client - Spoofed UI as part of a larger attack scenario" />
<LISTITEM value="Server - Spoofed specific user or computer over secure protocol" />
<LISTITEM value="Server - Spoofed random user or computer over secure protocol" />
<LISTITEM value="Client - Tampered trusted data that persists after restart" />
<LISTITEM value="Client - Tampered data that does not persist after restart" />
<!-- additional allowed values omitted for brevity -->
</ALLOWEDVALUES>
<DEFAULT from="value" value="Not Applicable" />
</FIELD>

We also want to further restrict the allowed value of EffectScope depending on the current value of Effect. If Effect is currently set to Spoofing, only the spoofing-related EffectScope values should be valid. If Effect is set to Tampering, only the tampering-related EffectScope values should be valid, and so on. We can accomplish this by adding WHEN clause elements to the FIELD definition (see Figure 8).

Figure 8 Adding a WHEN Clause to Restrict Allowed EffectScope Values

<FIELD
reportable="dimension"
type="String"
name="EffectScope"
refname="MSDN.SDL.Security.Effect.Scope">
<HELPTEXT>The scope of the effect of the security bug. This value is used to determine the default bug bar severity</HELPTEXT>
<ALLOWEDVALUES>
<!-- omitted for brevity -->
</ALLOWEDVALUES>
<DEFAULT from="value" value="Not Applicable" />
<WHEN field="MSDN.SDL.Security.Effect" value="Not a Security Bug">
<ALLOWEDVALUES>
<LISTITEM value="Not Applicable" />
</ALLOWEDVALUES>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect" value="Attack Surface Reduction">
<ALLOWEDVALUES>
<LISTITEM value="Not Applicable" />
</ALLOWEDVALUES>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect" value="Spoofing">
<ALLOWEDVALUES>
<LISTITEM value="Client - Spoofed trusted UI in common/default scenario" />
<LISTITEM value="Client - Spoofed trusted UI in specific other scenario" />
<LISTITEM value="Client - Spoofed UI as part of a larger attack scenario" />
<LISTITEM value="Server - Spoofed specific user or computer over secure protocol" />
<LISTITEM value="Server - Spoofed random user or computer over secure protocol" />
<LISTITEM value="Not Applicable" />
</ALLOWEDVALUES>
</WHEN>

<!-- Additional WHEN elements for the other STRIDE values omitted for brevity -->

</FIELD>

Now it’s time to implement the allowed values logic for the BugBarSeverity field. The logic for BugBarSeverity is slightly different from the Effect logic in that we don’t want the user to be able to directly set the value of the BugBarSeverity field. The whole point of implementing a bug bar like this is that the severity should reflect the characteristics of the vulnerability. If a user could just set the severity to any value he or she wanted, it would completely defeat the purpose.

Instead of creating a list of allowed values for BugBarSeverity, we will use WHEN fields to copy the appropriate value into the BugBarSeverity field, as determined by the bug bar we defined earlier, based on the current value of Effect. For example, the bug bar specifies that a spoofed trusted UI in a common or default scenario should be treated as an Important bug, so when Effect is “Client – Spoofed trusted UI in common/default scenario,” we copy “2 – Important” into BugBarSeverity (see Figure 9).

Figure 9 Implementing Value Logic for the BugBarSeverity Field

<FIELD
reportable="dimension"
type="String"
name="BugBarSeverity"
refname="MSDN.SDL.Security.Severity.BugBar">
<HELPTEXT>The suggested severity of the bug as determined by the security bug bar</HELPTEXT>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Not Applicable">
<COPY from="value" value="4 - Low"/>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Client - Spoofed trusted UI in common/default scenario">
<COPY from="value" value="2 - Important"/>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Client - Spoofed trusted UI in specific other scenario">
<COPY from="value" value="3 - Moderate"/>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Client - Spoofed UI as part of a larger attack scenario">
<COPY from="value" value="4 - Low"/>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Server - Spoofed specific user or computer over secure protocol">
<COPY from="value" value="2 - Important"/>
</WHEN>
<WHEN field="MSDN.SDL.Security.Effect.Scope" value="Server - Spoofed random user or computer over secure protocol">
<COPY from="value" value="3 - Moderate"/>
</WHEN>

<!-- additional WHEN clauses omitted for brevity -->

</FIELD>

You might be wondering why I’ve added number prefixes to the values, like “1 – Critical” and “2 – Important,” instead of just defining them as “Critical” and “Important.” The answer is that TFS automatically alphabetizes lists of allowed values, and without the number prefixes the choices would be displayed out of order and could confuse the user.

Also, the inherent MSF-Agile Severity field (Microsoft.VSTS.Common.Severity) values have the same number prefixes applied to them, so adding the prefixes to the BugBarSeverity fields reinforces the fact that there is a relationship between these two fields.

Speaking of the relationship between Severity and BugBarSeverity, it’s time to enforce that relationship in the template. The next step in the process is to constrain the value of the Severity field so that it’s at least as severe as the BugBarSeverity field. If the team has a specific business reason to make the actual severity higher than the value determined by the bug bar, that’s OK—we just don’t want it to work the other way.

To make this work, we use the same ALLOWEDVALUES technique we used to constrain the EffectScope field based on the value of the Effect field (see Figure 10).

Figure 10 Constraining Allowed Values for the Severity Field

<FIELD
name="Severity"
refname="Microsoft.VSTS.Common.Severity"
type="String"
reportable="dimension">
<HELPTEXT>Assessment of the effect of the bug on the project</HELPTEXT>
<ALLOWEDVALUES expanditems="true">
<LISTITEM value="1 - Critical"/>
<LISTITEM value="2 - High"/>
<LISTITEM value="3 - Medium"/>
<LISTITEM value="4 - Low"/>
</ALLOWEDVALUES>
<DEFAULT from="value "value="3 - Medium" />

<WHEN field="MSDN.SDL.Security.Severity.BugBar"value="1 - Critical">
<ALLOWEDVALUES expanditems="true">
<LISTITEM value="1 - Critical"/>
</ALLOWEDVALUES>
</WHEN>
<WHEN field="MSDN.SDL.Security.Severity.BugBar" value="2 - Important">
<ALLOWEDVALUES expanditems="true">
<LISTITEM value="1 - Critical"/>
<LISTITEM value="2 - High"/>
</ALLOWEDVALUES>
</WHEN>
<WHEN field="MSDN.SDL.Security.Severity.BugBar" value="3 - Moderate">
<ALLOWEDVALUES expanditems="true">
<LISTITEM value="1 - Critical"/>
<LISTITEM value="2 - High"/>
<LISTITEM value="3 - Medium"/>
</ALLOWEDVALUES>
</WHEN>
</FIELD>

There is one final change you need to make to the Bug work item: you need to add UI controls for the new fields we’ve added, as shown in the snippet below. Work item controls are defined in the witd:WITD/WORKITEMTYPE/FORM/Layout section of the document. It’s up to you where you place the new fields, but I suggest adding a new “Security” tab to the main TabGroup and adding the fields there:

<Tab Label="Security">
<Control Field Name="MSDN.SDL.Security.Effect" Type="FieldControl" Label="Effect "LabelPosition="Top" />
<Control Field Name="MSDN.SDL.Security.Effect.Scope" Type="FieldControl" Label="Scope" LabelPosition="Top" />
<Control Field Name="MSDN.SDL.Security.Severity.BugBar" Type="FieldControl" Label="Bug Bar Rating" ReadOnly="True" LabelPosition="Top" />
</Tab>

You are now finished editing the Bug work item definition. However, before you can use the new bug bar functionality, you have to import the modified process template definition back into Team Foundation Server. You have two choices for how to do this. You can either replace the existing MSF-Agile for Software Development template with the new template, or you can add the new template side-by-side with the previous version.

If you choose to replace the existing template, open the Process Template Manager, select the MSF-Agile for Software Development template and click the Delete button. Note that this is a non-reversible action. You won’t be able to get the original template back without reinstalling Team Foundation Server. Once you’ve deleted the existing MSF-Agile template, click the Upload button and select the folder containing the edited process template. The template will appear in the list as “MSF-Agile for Software Development” just as before, but now any future projects created from this template will have the bug bar functionality.

If you choose to add the new template side-by-side with the existing MSF-Agile template instead of replacing it, you’ll need to change the name of your new template so that it doesn’t conflict with the existing one. To do this, you need to make one more file edit. In your XML editor of choice, edit the file ProcessTemplate.xml, which can be found in the folder to which you originally downloaded the template. Change the value of the ProcessTemplate/metadata/name element to something like “MSF for Agile Software Development plus Bug Bar,” save the file and exit. Open the Process Template Manager, click the Upload button and select the folder containing the edited template. The new template will appear in the list with the name you’ve given it, and any future projects created from this template will include the bug bar.

Using the Bug Bar to Determine Exit Criteria

Of course, all the process template functionality in the world still won’t help you unless you have an organizational policy to back it up. The standard for internal development teams at Microsoft is that any security bug that falls above the “Low” category of the bug bar must be fixed before release. This standard is never relaxed, no matter how close the product is to release. This approach assures the objectivity of security bug triage, based solely on the possible effect and scope of the bug.

This is the reason we define a separate BugBarSeverity field instead of just constraining the value of the common Severity field based on EffectScope. From a strict security standpoint, we don’t care if the product ships with any severity Critical bugs as long as those bugs have no security effects. All we really care about is whether the bug has a BugBarSeverity higher than “4 – Low.” If so, that bug must be fixed before release.

Call to Action

Without a consistent, objective process for triaging security bugs, you will not be able to create secure applications. Using the Microsoft security bug bar is an excellent way to accomplish this, and it’s a key component of the Security Development Lifecycle, which has been proven to reduce vulnerabilities in software.

Additionally, if you are using Microsoft Team Foundation Server, you can easily add bug bar functionality to your team projects. This makes it even easier for your team to appropriately classify security bugs, even if the developers are not necessarily security experts.

As a final thought, I’d like to encourage you to download the MSF-Agile + SDL process template for Visual Studio 2010. This process template will be available for free at microsoft.com/sdl shortly after the release of Visual Studio 2010. It incorporates the bug bar described in this article, as well as many other features designed to help you create more secure software.                                                                  

Bryan Sullivan is a security program manager for the Microsoft Security Development Lifecycle team, where he specializes in Web application and .NET security issues. He is the author of “Ajax Security” (Addison-Wesley, 2007).

Thanks to the following technical expert for reviewing this article: Brian Harry