• Article

Automating JavaScript Testing with QUnit

Jörn Zaefferer | March 22, 2011

 

QUnit*** was born May 2008, out of the testrunner embedded into the jQuery core repository. It got its own name and documentation and a new home for the code. Late 2009 QUnit was refactored to be independent of jQuery, making it useful for testing all kinds of JavaScript frameworks and applications. Its assertion methods now follow the CommonJS assert specification. While QUnit can run in server-side or command line environments, it’s still most useful for testing JavaScript in the browser. This article explores how to write unit tests with QUnit, and how QUnit can help developing applications.***

* *

To get started, let’s look at a minimal QUnit testsuite, the Hello World of QUnit:

<!DOCTYPE html>
<link rel="stylesheet" href="qunit.css">
<script src="qunit.js"></script>
<script>
test("hello", function() {
    ok(true, "world");
});
</script>
<h1 id="qunit-header">QUnit Hello World</h1>
<h2 id="qunit-banner"></h2>
<ol id="qunit-tests"></ol>

This is a very minimalistic testsuite. It declares a doctype, includes the CSS and JS files for QUnit, defines another script element with a single QUnit test and assert, and a little bit of additional markup for QUnit to output testresults. The call to the test() function defines a test with the name of “hello”. QUnit will then run this test once the page has loaded. The second argument, a function passed to test(), contains the actual testcode to run, here a call to ok(). The first argument defines if the assertion passes or not, the second specifies a message to output. In this case, the boolean true will always pass, as we’re not yet testing anything really.

If we open this as a html file in a browser, we’ll see this output:

We can now set the assertion to always fail:

test("hello", function() {
    ok(false, "world");
});

The result will be this:

Now we see the hello test failing, along with the failing assertion. You’ll also notice in the screenshot the noglobals and notrycatch checkboxes and the text Rerun next to the testname. These are actually interactive elements. We’ll get to these later. For now, let’s write some actual tests.

Writing QUnit Tests

Let’s look at a more complete example:

<!DOCTYPE html>
<html>
<head>
    <title>QUnit Test</title>
    <link rel="stylesheet" href="qunit.css">
    <script src="qunit.js"></script>
    <script src="tests.js"></script>
</head>
<body>
    <h1 id="qunit-header">QUnit Test</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture">test markup</div>
</body>
</html>

We’ve added html, head and body elements, referenced an additional .js file and added some more markup for QUnit to interact with. The content of tests.js is this:

function format(string, values) {
    for (var key in values) {
        string = string.replace(new RegExp("\{" + key + "}"), values[key]);
    }
    return string;
}

test("basics", function() {
    var values = {
        name: "World"
    };
    equal( format("Hello, {name}", values), "Hello, World", "single use" );
    equal( format("Hello, {name}, how is {name} today?", values),
        "Hello, World, how is World today?", "multiple" );
});

We’re now testing a function called format, which expects a string template and an object with keys and values to replace within that template.

The test for that function uses the equal() assertion. It compares the first two arguments using JavaScript’s “==” operator. The first argument is the actual value, in this case being the output of calling format(). The second argument is the expected value; the formatted string.

Running it in the browser produces this:

As we can see, the implementation actually has a bug as it doesn’t replace a single key multiple times. Using the equal() assertion makes this easy to spot as we see both the actual and the expected value, a diff between the two, and (depending on the browser you use, works fine in Firefox and Chrome), even filename and line number of the bad assertion. Line 13 points to the second equal() assertion.

We can easily fix that issue by passing the global flag to the RegExp constructor:

new RegExp("\{" + key + "}", "g")

With that modification, both assertions pass. We could now extend the testsuite by also writing a test that involves a template with more than one key. Give that a try if you want. Up next: How to avoid one test breaking other tests.

Atomic

As long as we test only functions with no side effects, testing is really easy. It gets more trickier when code has side effects such as modifying the DOM, or introducing or modifying global variables. It’s certainly a reasonable goal to write code free of side effects, as that has advantages beyond just testing, but it’s not always possible. To keep tests atomic - independent of each other - QUnit provides a few tools that you can use.

DOM fixture

For tests that modify the DOM, we can use the #qunit-fixture element. We can put static markup in there and have each test use that or we can leave it empty and have each test append the elements it needs, without having to worry about removing them. QUnit will automatically reset the innerHTML property of the #qunit-fixture element after each test to the initial value. If jQuery is available, QUnit uses jQuery’s html() method instead, which also cleans up jQuery event handlers.

If we want to test a HTML5 placeholder polyfill, we’d start with an input element and a placeholder attribute:

<div id="qunit-fixture">
    <input id="input" type="text" placeholder="placeholder text" />
</div>

Now each test could select the #input element, modify it, and QUnit would reset it afterwards.

module

If we need more cleanup than just resetting the DOM, we can do that as part of each test. If we end up with multiple tests having to do the same cleanup, we can refactor that using the module() method. The primary purpose of module() is for grouping tests, e.g. multiple tests that test a specific method can be part of a single module. That helps when filtering tests (see Developing with QUnit below). For making tests atomic, we can use the setup and teardown callbacks module() provides:

module("core", {
    setup: function() {
        // runs before each test
    },
    teardown: function() {
        // runs after each test
    }
});
test("basics", function() {
    // test something
});

The setup callback is run before each test in this module, teardown is run after each test.

We can also use these callbacks to create objects and use them within tests, without having to rely on closures (or global variables) to pass them to the test. That works as both setup and teardown and the actual tests are called within a custom scope that is shared and cleaned up automatically.

Here’s some code to test a simple (and incomplete) library for handling monetary values:

var Money = function(options) {
    this.amount = options.amount || 0;
    this.template = options.template || "{symbol}{amount}";
    this.symbol = options.symbol || "$";
};
Money.prototype = {
    add: function(toAdd) {
        this.amount += toAdd;
    },
    toString: function() {
        return this.template
            .replace("{symbol}", this.symbol)
            .replace("{amount}", this.amount)
    }
};
Money.euro = function(amount) {
    return new Money({
        amount: amount,
        template: "{amount} {symbol}",
        symbol: "EUR"
    });
};

The code creates an object Money, with one default constructor for dollars, a factory method for euros, and two methods for manipulating and printing. Instead of creating new Money objects for each test, we use the setup callback to create objects and store them in the test scope:

module("Money", {
    setup: function() {
        this.dollar = new Money({
            amount: 15.5
        });
        this.euro = Money.euro(14.5);
    },
    teardown: function() {
        // could use this.dollar and this.euro for cleanup
    }
});

test("add", function() {
    equal( this.dollar.amount, 15.5 );
    this.dollar.add(16.1)
    equal( this.dollar.amount, 31.6 );
});
test("toString", function() {
    equal( this.dollar.toString(), "$15.5" );
    equal( this.euro.toString(), "14.5 EUR" );
});

Here two objects are created and stored on setup. The add test uses just one of them, the toString test uses both.The teardown callback isn’t necessary here, as there’s no need to remove the created Money objects.

Testing asynchronous code

We’ve seen that QUnit controls when to actually run tests, and that works fine as long as your code runs synchronously. Once code under test requires usage of asynchronous callbacks (e.g. due to timeouts or AJAX requests), you need to give QUnit feedback about that so that it stops running the next test and waits for you to tell it to go on.

The methods for that are simply called stop() and start(). Here’s an example:

test("async", function() {
    stop();
    $.getJSON("resource", function(result) {
        deepEqual(result, {
            status: "ok"
        });
        start();
    });
});

Here jQuery’s $.getJSON method is used to request data from a resource and then assert the result. For that deepEqual is used (instead of the previous equal), to check that the result is exactly what we expect.

As $.getJSON is asynchronous, we call stop(), then run the code, and at the end of the callback, call start() to tell QUnit to continue running tests.

Running asynchronous code without telling QUnit to stop would cause arbitrary results like failed (or passed) assertions showing up in other tests.

asyncTest

We can drop the call to stop() and use asyncTest() instead of test(). That makes it more obvious that a test runs asynchronous:

asyncTest("async2", function() {
    $.getJSON("resource", function(result) {
        deepEqual(result, {
            status: "ok"
        });
        start();
    });
});

expect

When testing callbacks, asynchronous or not, we can’t be sure if our callback will actually get called at some point. For that we can use expect(), to define the number of assertions we expect within a test. That way we can avoid a test that passes with just one good assertion, when there should’ve been two assertions.

asyncTest("async3", function() {
    expect(1);
    $.getJSON("resource", function(result) {
        deepEqual(result, {
            status: "ok"
        });
        start();
    });
});

Here, within an asyncTest, expect is called with an argument of 1, telling QUnit that this test should finish with exactly one assertion. When dealing with multiple asynchronous tests, adding the expect call makes sense even if there is just one code path for the test. If another test is broken and leaks assertions into this test, the expect call helps catching that problem.

async semaphore

If a test has multiple potential endpoints - multiple callbacks that run in random order - we can use QUnit’s built-in semaphore. By calling stop() as often as you call start() later on, QUnit will only continue to run once the internal counter increased by stop() is decreased back to zero by start().

test("async semaphore", function() {
    stop();
    stop();
    $.getJSON("resource", function(result) {
        equal(result.status, "ok");
        start();
    });
    $.getJSON("resource", function(result) {
        equal(result.status, "ok");
        start();
    });
});

More assertions

So far we’ve seen ok, equal and deepEqual in use. But QUnit provides a few more assertions. There’s strictEqual, which works the same as equal, except that it uses strict comparisons via JavaScript’s “===” operator. When comparing values of potentially different types, e.g. numbers and strings, it makes sense to use strictEqual, so that you don’t end up with a test where 0 equals "0" (the number zero vs the string zero).

For equal, deepEqual and strictEqual there are inverted counterparts: notEqual, notDeepEqual, and notStrictEqual. The same rules apply, it’s just the result that gets inverted.

raises

In addition to these, there’s the raises assertion (similar to throws in the CommonJS spec, but not using a reserved keyword). You pass a function as the first argument, which will get called by QUnit within a try-catch block. If the function throws an exception, the assertion passes, otherwise fails. You can also test the thrown exception using the second argument. It accepts a regular expression to test the exception, a constructor function to test with instanceof, or just a function that gets the exception passed as the first argument and returns a boolean for valid or invalid results.

test("raises", function() {
    function CustomError() {}
    raises(function() {
        throw new CustomError();
    }, CustomError, "must throw error to pass");
});

The example shows how to use raises along with testing for the constructor of the thrown exception, by passing a reference to the expected constructor as the second argument.

Custom assertions

It’s common for assertions to require more than just a call to equal(). A good way of refactoring test code is to extract custom assertion methods. To make the output consistent with other assertions, you can use QUnit.push() directly. Here’s an example, from the jQuery UI testsuite:

function domEqual( selector, modifier, message ) {
    var attributes = ["class", "role", "id", "tabIndex", "aria-activedescendant"];
    
    function extract(value) {
        var result = {};
        result.nodeName = value[0].nodeName;
        $.each(attributes, function(index, attr) {
            result[attr] = value.attr(attr);
        });
        result.children = [];
        var children = value.children();
        if (children.length) {
            children.each(function() {
                result.children.push(extract($(this)));
            });
        } else {
            result.text = value.text();
        }
        return result;
    }
    var expected = extract($(selector));
    modifier($(selector));
    
    var actual = extract($(selector));
    QUnit.push( QUnit.equiv(actual, expected), actual, expected, message );
}

This method is used by tests to verify that the destroy-method is properly implemented, leaving the element after calling destroy in the same state as it was before it was initialized. It replaces comparing the innerHTML property, as that produced way too many false negatives. Instead this method uses a list of attributes to compare, and recursively goes through all child elements, getting the text content of elements without children.

At the end, the two results are compared using QUnit.equiv, the deep comparsion implmentation used by deepEquals. Both the actual and expected results are also passed to QUnit.push, along with forwarding the message argument. By using QUnit.push directly, QUnit can output correct file names and line numbers. If we used deepEqual instead, the line number would point to our custom assertion calling deepEqual, instead of the code calling the custom assertion.

Here’s an example from the jQuery UI autocomplete testsuite for using the method:

test("destroy", function() {
    domEqual("#autocomplete", function() {
        $("#autocomplete").autocomplete().autocomplete("destroy");
    });
})

The custom assertion method is called with a selector and a callback. The selector specifies what element to test; the callback does the actual modification. The assertion method takes snapshots before and after applying the modifier, then comparing the result.

Developing with QUnit

Between writing tests we spend a lot of time looking at test results, trying to figure out why a particular test failed to run or why a test that was supposed to fail passed instead.

QUnit provides tools that help at this step. We’ve already seen the detailed test output that includes actual and expected values, a diff between the two and file names and line numbers.

When working on one particular test, it helps to rerun only that test instead of the full testsuite. It can make a difference in the time to run the tests (less tests = less time), but it can also help to focus on one particular issue when there are other unrelated failing tests.

To rerun a single test, click on the Rerun link next to the name of the test. It’ll open the same testsuite again, but adding a “?filter=name-of-test” to the URL. The name will be URL-encoded, so may look a bit odd. The useful part of having the address change is that you can just reload the page to test again with the same filter, or navigate back to run the full testsuite.

Here the QUnit testsuite is filtered to run only the raises test from the assertions module.

You can also modify the address manually. If we set filter=assertions, the testsuite will run all the tests that contain “assertions” in their name:

Debugging tools

We’ve seen the noglobals and notrycatch checkboxes in the QUnit header. Clicking those sets a special filter.

noglobals

When noglobals is checked, QUnit enumerates all properties on the window object before running a test, and then compares that list to the window object after finishing the current test. If there’s a difference, the test fails, with an additional assertion outputting the introduced or missing global property. This makes it easy to spot global variables that are introduced by accident.

If we modify our first test and remove the var keyword in front of the values variable, then rerun the testsuite with noglobals checked, we get this:

Sometimes code has to introduce global variables; therefore the feature is off by default. But it can help to spot mistakes by enabling it every once in a while.

notrycatch

The other checkbox is notrycatch. This tells QUnit to run test callbacks outside of the usual try-catch block. When a test fails with an exception, QUnit won’t be able to catch it, therefore the testsuite will stop running. This becomes useful when debugging a particular exception, epsecially in browsers with poor built-in JavaScript debugging tools. Especially in IE6, an unhandled exception provides much more useful information then a catched exception. Even if rethrown, due to JavaScripts rather poor exception handling, the original stacktrace is lost in most browsers.

This is obviously a very special-purpose tool. But once you are hit by an exception that you can’t trace back to actual code, this can be extremely useful.

More efficient TDD

QUnit is a great tool for test-driven development (TDD, see other Script Junkie articles) as-is, but with a little tweak it becomes even more useful in providing useful results much faster.

Consider a larger testsuite that takes ten seconds to run. A test somewhere at the end fails after a change you did, correctly so. You implement the missing functionality, rerun the test, wait ten seconds for the result, and still get a failure. Ten seconds can easily become 30 seconds, 1 minute or worse. But even with 10 seconds per run, TDD is starting to fall apart. Running just parts of the testsuite using filters isn’t always an option, as you want verification that you didn’t break anything else with your change. Optimizing a testsuite to run faster can be a challenge in itself, as it’s hard to tell if a change to the testsuite actually breaks something.

What you can do here is check the “Hide passed tests” checkbox. Once checked, QUnit won’t show any tests it currently runs or that passed, instead you’ll see only failing tests.

With that in place, another QUnit feature becomes much more obvious and useful. Under the hood, QUnit keeps track of which tests, in a previous run, failed. It uses sessionStorage for that, if supported (in pretty much every modern browser). The next time you run the suite, it’ll run the failing tests first, while maintaining the same output order. Without “Hide passed tests” checked, that is usually not obvious, as the failing test gets lost somewhere in the long list of passing tests. But once checked, passed tests and the “Running test [...]” placeholders are hidden.

With that, rerunning the 10-second suite will instantly output the result of the failing test, and we can go ahead and continue working on the code. Once we come back we can still check the end result of the run to see if there were any additional failures. If not, we just reload to run again, and get instant feedback on the test we care about.

Testing multiple jQuery versions

If you’re building jQuery plugins or some other form of library or framework that you want to test against multiple versions of a dependency, here’s a useful technique to do that without requiring any modifications of your files or serverside code. Instead of referencing jQuery directly, create a script that checks the query string and then loads the specified jQuery version. Scott González came up with this technique for the AmplifyJS testsuite. Here’s a slightly modified version used the by Validation Plugin:

(function() {
var parts = document.location.search.slice( 1 ).split( "&" ),
    length = parts.length,
    i = 0,
    current,
    version = "";
for ( ; i < length; i++ ) {
    current = parts[ i ].split( "=" );
    if ( current[ 0 ] === "jquery" ) {
        version = current[ 1 ];
        break;
    }
}
if (version) {
    version = "-" + version;
}
document.write( "<script src='../lib/jquery" + version + ".js'></script>" );
})();

The actual script include happens synchronously with document.write, which works fine in this context.

In order to easily rerun the testsuite with these different versions, the plugin’s testsuite adds a few links to the header:

That way it becomes a matter of clicking a few links to run the same suite against multiple versions of jQuery. The same technique could be applied to test against other dependencies as well.

Conclusion

We’ve seen how QUnit provides the tools both for writing and refactoring tests, and developing frameworks and applications more efficiently. Even if you end up using a different testing tool, you should look out for the described features, as they make testing much more efficient.

 

About the Author

Jörn is a Java engineer living in Cologne, Germany. He’s a Development Lead and contributed his accordion and autocomplete plugins to jQuery UI. He has also been a driving force of the jQuery core development process, pushing out many of the 1.0.x releases. He’s responsible for completely rebuilding the jQuery test suite, now known as QUnit, and writing a large number of the test cases.

Find Jörn on: