• Article

2009

Volume 24 Number 0

Extreme ASP.NET Makeover - Testing

By James Kovacs | 2009

Welcome back for the second installment of the multipart article series: Extreme ASP.NET Makeover. In Part 1, we started on our journey to improve an existing Web application, the ScrewTurn Wiki, and turn it into a rich Internet application by applying the latest Web techniques. I talked about brownfield basics—four fundamental software engineering principles that every project should adopt, regardless of methodology:

  1. Version control
  2. Issue tracking
  3. Automated, self-contained builds
  4. Automated testing

Part 1 focused on the first three of these principles. This article, Part 2, will focus on the remaining principle - automated testing.

Of Tightropes and Tests

Applying a strong suite of automated tests provides a codebase with a safety net. Developers can modify code protected by tests in relative safety. Method implementations can be changed, class hierarchies refactored, and code improved in a myriad of ways. Automated tests can be run on the code at any time to verify that existing functionality is not broken. Existing automated tests act to prevent regressions from entering the codebase. (A regression is a feature that previously worked, but does not anymore.)

Now why would a development team want to modify working code? There are many reasons:

  • A bug needs to be fixed and you want to fix it without breaking other functionality.
  • You think of a better or more efficient method of implementing a feature.
  • You want to pay down some technical debt accrued by the team.

(Technical debt is a metaphor created by Ward Cunningham to describe the gradual accumulation of friction in a codebase due to quick-and-dirty programming. As technical debt builds, the difficulty in working with a codebase increases. You can reduce this debt by refactoring the code to eliminate duplication, increase cohesion, decrease coupling, etc. Your chances of refactoring code safely increases, thereby reducing technical debt, if you have a good automated test suite acting as a safety net.)

Just like a safety net used when walking tightropes, a suite of automated tests is only as good as its coverage. Having a safety net isn't much use if you fall outside of the net’s perimeter or if the holes in the net are big enough to let you fall through. The same holds true for automated tests. If you are modifying code that isn't tested, you are walking a tightrope without a safety net.

Testing Landscape in an ASP.NET World

We will focus on developer-oriented testing, including unit, integration, and acceptance testing. These terms are often confused such that most "unit tests" are actually integration tests. Developers think that they're writing unit tests because they're using a unit testing framework. What we, as developers, call "unit testing frameworks" are in reality "testing frameworks" that can be used to write a wide variety of test types.

If most developers are actually doing integration testing, what is unit testing? Unit testing involves testing a single component, which in object-oriented terms is a class. With integration testing, we are testing a set of components working together.

There are a wide variety of testing frameworks available for the Microsoft .NET Framework. The most commonly used ones are NUnit, MbUnit, xUnit, and MSTest. All are .NET ports of JUnit, the original Java testing framework. Actually the original unit testing framework was SUnit, written by Kent Beck, for Smalltalk, but wasn't widely distributed. Beck went on to co-write JUnit with Erich Gamma and JUnit did gain widespread acceptance.

ScrewTurn Wiki already uses NUnit for unit and integration testing and we will continue to use NUnit throughout the series. The unit and integration tests apply to business, data access, workflow, and similar logic. They do not interact with the HTML and JavaScript of a Web page. These tests provide us with a safety net when working with the core application code. We are going to examine ScrewTurn Wiki's unit and integration tests in detail in a future article when we talk about refactoring the core application code. In the next few articles, we will be updating and improving ScrewTurn Wiki's HTML, CSS, and JavaScript. We need a safety net here too and so we turn our focus in this article to automated acceptance tests around the Web user interface.

Automated Acceptance Testing

Acceptance testing is black-box, end-to-end testing of a system to ensure that it meets the needs of the end user. Acceptance tests are often typified by long, verbose Word documents with step-by-step manual instructions, which are performed and verified by the QA department or end users. Here is a simple manual acceptance test:

Verify Administrative Portal Requires Admin Credentials

  1. Browse to the main page at https://server/Default.aspx.
  2. If currently logged in, click Logout.
  3. Verify that you are accessing the site as a guest. ("Welcome guest" should be displayed in the top right.)
  4. Browse to https://server/Admin.aspx.
  5. Verify that you are re-directed to https://server/Login.aspx and prompted for credentials.

A typical system will have dozens to hundreds of these tests. Manual testing is repetitive, monotonous, time-consuming, and error-prone. The immediate question that should jump to mind is, "How difficult is it to automate these acceptance tests?" As it turns out, it is a lot easier than you might otherwise think.

Introducing WatiN

WatiN is a Web application testing framework written in C#. It is an acronym for Web Application Testing in .NET and was inspired by WatiR - Web Application Testing in Ruby. To use WatiN, you simply reference WatiN.Core in your test project and start writing test code. It integrates with all of the major .NET testing frameworks. Here is some NUnit code using WatiN:

[Test]
public void CanBrowseToMicrosoft() {
    using(var browser = new IE()) {
        browser.GoTo("https://www.microsoft.com/");
        Assert.AreEqual("Microsoft Corporation", browser.Title);
    }
}

In the code above, IE is a class defined in WatiN.Core. When an instance of IE is created, an actual instance of Internet Explorer is launched. WatiN automates the browser, instructing it to go to a particular URL and retrieve the title of the resultant page. WatiN can also click on buttons and links, fill in form fields, hover over elements, and just about anything else a real user can do. Since WatiN is automating a real browser, you have real JavaScript and Asynchronous JavaScript + XML (AJAX) support. With WatiN, you can automate complex user interactions that traverse multiple pages, such as adding or editing a page on a wiki, buying an item (including a multi-step checkout) on an e-commerce site, or completing a multi-page survey on a polling site. Rather than write isolated simple tests, you can actually automate business-relevant workflows. Since you're running the acceptance tests through a testing framework, you can create a clean database and/or site with default content before every test run, which maximizes your ability to create a reproducible set of acceptance tests that can be run on demand without user interaction.

In addition to automating Internet Explorer, WatiN 2.0 (which is currently in beta) can automate Mozilla Firefox v2/v3 and has very preliminary support for Google Chrome. Eventually you will be able to do some cross-browser testing with WatiN, though not cross-platform because WatiN directly automates the browser, which must be running on Windows. We will be focusing on Internet Explorer since WatiN's support for that browser is the most mature.

Setting Up WatiN

Wouldn't it be wonderful if software just worked right out of the box every time? WatiN is a bit trickier than most. In getting WatiN running on your system, you have two hurdles to overcome. WatiN is interacting with Internet Explorer through a COM automation API. WatiN is able to hide almost all of the COM-related details from you except for one thing: the threading model. The Internet Explorer COM automation API assumes that you're executing in the single-threaded apartment (STA). This is COM-speak for only allowing one thread to interact with a component. Some test runners use a multithreaded apartment (MTA), which means any thread can access the component. For example, NUnit and MbUnit test runners both use the MTA by default. xUnit, MSTest, TestDriven.NET, and JetBrains ReSharper test runners use the STA. Note that use of a STA or MTA thread for running tests is dependent on the test runner rather than the test framework. TestDriven.NET running NUnit tests will run them on a STA thread, but NUnit GUI running NUnit tests will run them on a MTA thread. To confuse matters even more, MbUnit will override the test runner so TestDriven.NET running MbUnit tests will run them on a MTA thread! This is a long way of saying that you need to run WatiN tests on an STA thread and you better set it yourself to make sure that WatiN can successfully communicate with Internet Explorer. For NUnit, we add an application configuration file called <TestAssemblyName>.dll.config (which in our case is called WebApplication-Test.dll.config) to our test project, as shown in Figure 1.

Figure 1 WebApplication-Test.dll.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <sectionGroup name="NUnit">
      <section name="TestRunner" 
               type="System.Configuration.NameValueSectionHandler"/>
    </sectionGroup>
  </configSections>

  <NUnit>
    <TestRunner>
      <add key="ApartmentState" value="STA" />
    </TestRunner>
  </NUnit>

</configuration>

On the file properties for WebApplication-Tests.dll.config, shown in Figure 2, change the Copy to Output Directory to "Copy if newer."


Figure 2 File Properties View

If you are using another test framework, you can find information on the WatiN Website on how to properly configure the apartment state.

Just one more change and we'll be able to execute our WatiN tests. Internet Explorer 7 and 8 on Windows Vista or later run in Protected Mode by default. Protected Mode prevents Internet Explorer (and hence any Web content running inside it) from modifying your system. It also hinders communication between Internet Explorer and WatiN. The easiest way to disable Protected Mode for a site is to add it to your Trusted sites list. As shown in Figure 3 , I have added https://www.microsoft.com to the Trusted sites list to run our simple WatiN, as well as https://localhost , since we'll be using a local Webserver to host our test site.


Figure 3 Trusted Sites Screen

For more detailed information on WatiN and Internet Explorer Protected Mode, see my blog post entitled "Running WatiN Tests on Vista". For more information on Internet Explorer Protected Mode, check out "Understanding and Working in Protected Mode Internet Explorer" in the MSDN Library.

With the test runner's COM apartment state set to STA and Internet Explorer Protected Mode disabled for https://www.microsoft.com, we can execute the simple WatiN test from above which browses to the Microsoft home page and verifies the page title.

You can learn more by viewing the following video clip.

 

 

Using WatiN to Test a Local Web site

Now that we have successfully run a simple WatiN test against a remote Web site, we turn our attention to using WatiN to test ScrewTurn Wiki. We won't want to deploy ScrewTurn Wiki to a server every time we want to run our acceptance test suite using WatiN. Instead, we will use the ASP.NET Development Server (WebDev.WebServer.exe), which is better known by its code name "Cassini." It is installed with .NET Framework 2.0 and above in C:\Windows\Microsoft.NET\Framework\v2.0.50727\. Another option would be to use a local IIS instance, but the ASP.NET Development Server has the nice feature that it can be started and/or stopped by a normal user.

We want to start ASP.NET Development Server before any WatiN tests are run and stop it once testing is complete. Most test frameworks have a method for specifying code to run before and after any tests. In NUnit 2.4.8, we use the SetUpFixture attribute on a class to mark it as containing setup/teardown code for all tests in the same namespace, as shown in Figure 4.

Figure 4 The SetUpFixture Attribute in NUnit

using System.Configuration;
using NUnit.Framework;

namespace ScrewTurn.Wiki.Tests {
    [SetUpFixture]
    public class WebServerRunner {
        private WebServer webServer;

        [SetUp]
        public void StartWebServer() {
            var absolutePathToWebsite = ConfigurationManager.AppSettings["absolutePathToWebsite"];
            webServer = new WebServer(absolutePathToWebsite, 9999);
            webServer.Start();
        }

        [TearDown]
        public void StopWebServer() {
            webServer.Stop();
            webServer = null;
        }
    }
}

The WebServer class is a helper class that encapsulates the ASP.NET Development Server. It needs the physical path to the compiled Web site and a fixed port. We need a fixed port, such as 9999, so that we can direct WatiN to browse to https://localhost:9999 in our tests.

ASP.NET Development Server on Windows Vista and Later

The ASP.NET Development Server only listens for requests on IPv4, not IPv6. If you are running Windows Vista or Windows Server 2008, requests for "localhost" will resolve to the IPv6 address of ::1 by default rather than the IPv4 address of 127.0.0.1 and the WatiN tests will fail. To resolve this issue, comment out the IPv6 localhost address in your C:\Windows\System32\drivers\etc\hosts file. This is the line with "::1 localhost".

The Start method for WebServer formats the command line arguments for WebDev.WebServer.exe and launches it using a System.Diagnostics.Process object:

public void Start() {

    webServerProcess = new Process();

    const string webDevServerPath = 
    @"c:\Windows\Microsoft.NET\Framework\v2.0.50727\WebDev.WebServer.exe"";

    string arguments = string.Format("/port:{0} /path:\"{1}\" /vpath:{2}", 
    port, physicalPath, virtualDirectory);

    webServerProcess.StartInfo = new ProcessStartInfo(webDevServerPath, arguments);

    webServerProcess.Start();

}

At the end of the test run, the Stop method terminates the Cassini process:

public void Stop() {
    if(webServerProcess == null) {
        throw new InvalidOperationException("Start() must be called before Stop()");
    }
    webServerProcess.Kill();
}

NOTE: You might receive a strange error message from the ASP.NET Development Server stating that "The directory '<PATH> /vpath:' does not exist", as we see in Figure 5. This is a bug in the ASP.NET Development Server which causes it to fail if your physical path has a trailing \. To bypass this bug, simply remove the trailing slash in your configuration file or code defensively using physicalPath.TrimEnd(\\) on any physical path intended for the ASP.NET Development Server as I did in the constructor for ScrewTurn.Wiki.Tests.WebServer.


Figure 5 ASP.NET Development Server Showing Error Message

Smoke Tests

Our goal in writing acceptance tests is to ensure that end user functionality continues to work as expected, as we add features or refactor existing code for better maintainability. Before getting into complex workflows, we will write some simple smoke tests using WatiN to ensure that ScrewTurn Wiki's basic functionality is working. (A smoke test is a preliminary test designed to show simple and obvious failures, such as an error in the web.config file resulting in the site being inaccessible.) Our smoke tests include the actions of:

  • Browsing to the main page
  • Logging into the site
  • Browsing the admin pages redirects to the login page
  • Accessing a nonexistent page redirects to a page not found information page

Note that you don't have to think of everything immediately, but can gradually add tests as you build your test suite.

We'll start out with testing whether a user can log into the site as an administrator. (By default, ScrewTurn Wiki's administrative account is admin/password.) The WatiN script should be fairly self-explanatory. You are finding page elements, such as the Login link, and then performing actions, such as clicking the Login link via its Click method, as shown in Figure 6.

Figure 6 WatiN CanLogIntoSite Test Script

[Test]
public void CanLogIntoSite() {
    using(var browser = new IE()) {
        browser.GoTo("https://localhost:9999");
        browser.Link(Find.ByTitle("Login")).Click();
        browser.TextField(Find.ByTitle("Type here your Username")).TypeText("admin");
        browser.TextField(Find.ByTitle("Type here your Password")).TypeText("password");
        browser.Button(Find.ByValue("Login")).Click();
        var username = browser.Link(Find.ByTitle("Go to your Profile")).Text;
        Assert.That(username == "admin");
    }
}

The only slightly confusing part might be how to find the page elements. (e.g., How did I know to find the username textbox by looking for its title, "Type here your Username"?) You need your browser of choice and a tool or two, such as Internet Explorer and the Internet Explorer Developer Toolbar or Firefox and Firebug/Web Developer Toolbar.

Finding by selecting an element's title isn't recommended because it is more likely to change. For example, the text really should say "Type your username here." If we correct the grammar, however, we break the test. I wanted to create the smoke tests before modifying the site, which is why I left the text as it is. A better option for identifying page elements is by ID. (You can find elements by ID using Find.ById("id") instead of Find.ByTitle("title"). The Find class contains a wide variety of static methods for simplifying the location of page elements.) So while WatiN can automate any site, you can make your tests more robust in the face of change (and language translations) by providing IDs for elements of interest.

The following video clip illustrates the use of the Web Developer Toolbar.

 

Eliminating Repetition from Tests

You'll notice a lot of potential for repetition in the CanLogIntoSite test (Figure 5).

  • Every WatiN test will have a URL starting with https://localhost:9999.
  • Many tests will require the user to be logged in.
  • Multiple tests might want to verify the currently logged in user.

As you can imagine, we'll see similar repetition as we write more tests. Let's apply the principle of Don't Repeat Yourself (DRY) to our WatiN test.

Let's apply the principle of Don't Repeat Yourself (DRY) to our WatiN test.

[Test]
public void CanLogIntoSite() {
    using(var browser = new IE()) {
        browser.GoTo(PageUrl.Default);
        browser.LoginAsAdmin();
        Assert.That(browser.IsLoggedInAs("admin"));
    }
}

I've created a PageUrl class that is simply a Uri list for pages in the site. I've also created some extension methods on WatiN's Browser class to script out common actions, as Figure 7 shows.

Figure 7 Extension Methods On WatiN’s Browser Class

public static void LoginAsAdmin(this Browser browser) {
    if(browser.IsLoggedInAs("admin")) {
        return;
    }
    browser.Link(Find.ByTitle("Login")).Click();
    browser.TextField(Find.ByTitle("Type here your Username")).TypeText("admin");
    browser.TextField(Find.ByTitle("Type here your Password")).TypeText("password");
    browser.Button(Find.ByValue("Login")).Click();
}
public static bool IsLoggedInAs(this Browser browser, string expectedUsername) {
    var username = browser.Link
(Find.ByTitle(title => title == "Go to your Profile" || 
title == "Select your language")).Text;
    return username == expectedUsername;
}

Not only am I encouraging re-use with my extension methods, I can more readily ascertain what my CanLogIntoSite test does by using concise, meaningful names for the helper methods. I have also eased my maintenance burden because when I do add IDs to the username/password textboxes and correct the grammar, I have minimized the number of changes I will need to fix my tests.

The following video clip shows the smoke tests being run.

 

Acceptance Tests

Acceptance tests using WatiN are simply longer versions of smoke tests. Figure 8 shows an acceptance test to verify that we can create a new page in our wiki.

Figure 8 Acceptance Test

[Test]
public void CanCreateNewWikiPage() {
    string pageName = string.Format("TestPageName-{0}", Guid.NewGuid());
    const string pageTitle = "TestPageTitle";
    using(var browser = new IE()) {
        browser.GoTo(PageUrl.Default);
        browser.LoginAsAdmin();
        browser.CreateWikiPage(pageName, pageTitle);
        Assert.That(browser.WikiPageTitle(), Is.EqualTo(pageTitle));
        Assert.That(browser.Url, Text.Contains(pageName));
    }
}

I created an extension method BrowserExtensions.CreateWikiPage to encapsulate the logic of creating a new page.

public static void CreateWikiPage(this Browser browser, string pageName, string pageTitle) {
    browser.Link(Find.ByTitle("Create a new Page")).Click();
    browser.TextField(Find.ByTitle("Type here the name of the page")).TypeText(pageName);
    browser.TextField(Find.ByTitle("Type here the title of the page")).TypeText(pageTitle);
    browser.Button(Find.ByValue("Save")).Click();
}

Another extension method simplifies finding a wiki page's title; not the contents of the <title/> tag, but the contents of the <h1 class="pagetitle"/> tag.

public static string WikiPageTitle(this Browser browser) {
    return browser.ElementWithTag("h1", Find.ByClass("pagetitle")).Text.Trim();
}

The following video clip shows the build process with acceptance tests.

 

As we write more acceptance tests, the BrowserExtensions class may become unwieldy in size. If this happens, I can easily refactor the code into BrowserLookupExtensions (for finding information in the page) and BrowserScriptingExtensions (for manipulating the page). I could separate extension methods by public and admin, particular function, or any other means to make the code more tractable.

Conclusion

As we have seen, an automated test suite provides the development team with a safety net. The smoke tests verify basic site functionality (e.g., the site is accessible) while the more detailed acceptance tests verify business-relevant functionality (e.g., visitors can update wiki pages). The automated test suite helps to ensure that working code continues to work as expected. It provides the team greater freedom and confidence to improve the codebase and eliminate technical debt. In Part 3, we will turn our attention to XHTML/CSS improvements and use our smoke and acceptance tests to ensure that we are not breaking the UI during our modifications.