Test Run
API Test Automation in .NET
James McCaffrey
Code download available at:TestRun0411.exe(124 KB)
Contents
The System Under Test
The API Test Automation
Discussion
The most fundamental type of software test automation is automated API testing. API testing essentially entails testing the individual methods that make up a software system rather than testing the overall system itself. API testing is also called unit testing, module testing, component testing, and element testing. Technically the terms are quite different, but in casual usage you can think of them as having roughly the same meaning. The idea is that you must make sure the individual building blocks of your system work correctly because if they don't the system as a whole cannot be correct. API testing is absolutely essential for any software system.
In this month's column I will show you key techniques and principles of automated API testing in a .NET environment. The best way to illustrate where I'm headed is with two screen shots. Figure 1 shows a dummy Windows®-based application program that is the overall system which uses the library I'm testing. In testing we often call the program or API being tested the system under test (SUT) to distinguish it from the test harness system.
Figure 1** StatCalc App **
The StatCalc application calculates the mean of a set of integers. Behind the scenes, the StatCalc application references my MathLib.dll library which houses a class that contains the ArithmeticMean, GeometricMean, and HarmonicMean methods. You might recall from an introductory statistics class that the arithmetic mean is a normal average, the geometric mean is an average of ratio values, and the harmonic mean is an average of rate values. For this example, I want to test only these three methods from the library, not the whole StatCalc application which includes additional code for the user interface.
In order to manually API test a single test case in this scenario, I would be required to create a small tester program, copy and paste the source code of one of the methods I'm testing from the source code for MathLib.dll into my tester program, hardcode some input arguments to the method, run the program, visually examine the return output, somehow determine if the result was correct or not, and record this information somewhere, for example in an Excel spreadsheet. I would need to do this several thousand times to even begin to feel confident that I understand the behavior of the methods I'm testing. And worst of all, every time the code in the system under test or its underlying DLL changed, I'd have to start all over. A much better testing strategy is to use the powerful capabilities of the .NET environment to write automated API tests. Figure 2 shows a sample run of a program that does just that.
Figure 2** Test Program Output **
In this example, the test program is a console application that reads test case data (test case ID, method to test, input arguments, and expected result), calls the specified method, and compares the actual result with an expected result to determine whether the test case passed or failed. In the sections that follow I will briefly examine the system under test so you'll know what I'm testing and how I'm testing it. I'll present and explain in detail the test program that generated the output shown in Figure 2, and I'll describe how you can extend the techniques presented here to meet your own needs.
The System Under Test
One of the most fundamental principles of software testing is that you must understand the system under test in order to test it. And the best way to understand an SUT is to examine its source code. Figure 3 lists the core of the code for the dummy system under test that calls the methods to verify.
Figure 3 Code for the System Under Test
Private Sub Button1_Click( _ ByVal sender As System.Object, ByVal e As System.EventArgs) _ Handles Button1.Click Dim sVals() As String sVals = TextBox1.Text.Trim.Split(" ") Dim iVals(sVals.Length - 1) As Integer For I As Integer = 0 To sVals.Length - 1 iVals(I) = Integer.Parse(sVals(I)) Next If RadioButton1.Checked Then Dim M As New MathLib.Methods TextBox2.Text = M.ArithmeticMean(iVals).ToString("F4") ElseIf RadioButton2.Checked Then Dim M As New MathLib.Methods TextBox2.Text = M.GeometricMean(iVals).ToString("F4") ElseIf RadioButton3.Checked Then MessageBox.Show("Not yet implemented") End If End Sub
Let me emphasize that I am definitely not using good coding techniques here so that I can keep the example as short and clean as possible (for example, I've made no attempt to handle poor input). The application has a Project Reference to file MathLib.dll, which you'll see in a moment. Notice that the application is not ready for the HarmonicMean method yet. Software testing takes place in a highly dynamic environment and, consequently, you have to expect incomplete implementations will be prevalent throughout the development and testing process.
Next, let's take a look at the source code in the MathLib.dll file that implements the key methods used by the application. Figure 4 lists this code. Again, let me emphasize that this source code is presented for illustrative purposes only.
Figure 4 MathLib.dll
Imports System Public Class Methods Public Function ArithmeticMean( _ ByVal ParamArray Vals() As Integer) As Double Dim Sum As Integer = 0 For Each V As Integer In Vals Sum = Sum + V Next Return Sum / Vals.Length End Function ' ArithmeticMean() Private Function NthRoot( _ ByVal X As Double, ByVal N As Integer) As Double Return Math.Exp(Math.Log(X) / CType(N, Double)) End Function Public Function GeometricMean( _ ByVal ParamArray Vals() As Integer) As Double Dim Product As Double = 1.0 For Each V As Integer In Vals Product = Product * V Next Return NthRoot(Product, Vals.Length) End Function ' GeometricMean() Public Function HarmonicMean( _ ByVal ParamArray Vals() As Integer) As Double ' not implemented yet Return 0.0 End Function ' HarmonicMean() End Class
In order to write test automation you must understand how to call each method being tested. And as you'll see in a moment, additional information is needed in order to create test case expected values. Here, each of the three methods accepts a variable number of int parameters and returns a single double value. Observe that both ArithmeticMean and GeometricMean are instance methods, so they are called from an object context. If I had implemented them as Shared methods, then I'd call them from a class context. The class contains a private helper method called NthRoot that's used by GeometricMean.
In general, API test automation does not test private helper methods because it is believed that any problems in them will be exposed by calling the methods that rely on the helpers. As with every general rule there are exceptions, and depending on the time and resources you have for your testing effort, you may want to test private methods, too (it can be very beneficial).
The API Test Automation
The basic API test harness is surprisingly short and is shown in Figure 5. The harness is a Visual Basic® .NET console application, often the type of program best suited for test automation. Console applications integrate easily into legacy test automation systems and can be manipulated easily in a Windows environment. As a general rule, you should write your test automation using the same language used by the system under test. If you use a different language you could run into cross-language issues (different interpretations of fundamental data types, for example). Before .NET this used to be a big problem but my recent experience at Microsoft has shown that test engineers can easily migrate among C#, Visual Basic .NET, and other .NET-compliant languages due to the unifying influence of the .NET Framework.
Figure 5 The Test Automation Harness
Imports System Imports System.IO Module Module1 Sub Main() Try Dim FS As New FileStream("..\TestCases.txt", FileMode.Open) Dim SR As New StreamReader(FS) Dim Line, CaseID, Method As String Dim Tokens(), TempInput() As String Dim Expected As String Dim Actual As Double Console.WriteLine() Console.WriteLine("CaseID Result Method Details") Console.WriteLine("=============================================") Console.WriteLine() Line = SR.ReadLine() While Line <> Nothing Tokens = Line.Split(":") CaseID = Tokens(0) Method = Tokens(1).Trim() TempInput = Tokens(2).Split(" ") Expected = Tokens(3) Dim Input(TempInput.Length - 1) As Integer For I As Integer = 0 To Input.Length - 1 Input(I) = Integer.Parse(TempInput(I)) Next Dim M As New MathLib.Methods If Method = "ArithmeticMean" Then Actual = M.ArithmeticMean(Input) If Actual.ToString("F4") = Expected Then Console.WriteLine(CaseID & " Pass " & Method & _ " actual = " & Actual.ToString("F4")) Else Console.WriteLine(CaseID & " *FAIL* " & Method & _ " actual = " & Actual.ToString("F4") & _ " expected = " & Expected) End If ElseIf Method = "GeometricMean" Then Actual = M.GeometricMean(Input) If Actual.ToString("F4") = Expected Then Console.WriteLine(CaseID & " Pass " & Method & _ " actual = " & Actual.ToString("F4")) Else Console.WriteLine(CaseID & " *FAIL* " & Method & _ " actual = " & Actual.ToString("F4") & _ " expected = " & Expected) End If ElseIf Method = "HarmonicMean" Then Console.WriteLine(CaseID & " " & _ Method & " Not yet implemented") Else Console.WriteLine(" unknown method") End If Line = SR.ReadLine() End While ' test case loop Console.WriteLine() Console.WriteLine("=============== end test run ================") SR.Close() FS.Close() Console.ReadLine() Catch ex As Exception Console.WriteLine(ex.Message) Console.ReadLine() End Try End Sub End Module
After creating a new Visual Basic .NET console application project using Visual Studio® .NET and adding a Project Reference to the MathLib.dll library, my first step was to create some test cases. I decided to use a simple text file. The following code displays the test case file that produced the output shown in Figure 2.
0001:ArithmeticMean:2 4 8:4.6667 0002:ArithmeticMean:1 5:3.0000 0003:ArithmeticMean:1 2 4 8 16 32:10.5000 0004:GeometricMean :1 2 4 8 16 32:6.6569 0005:GeometricMean :0:0.0000 0006:GeometricMean :2 4 8:4.0000 0007:HarmonicMean :2 4 8:3.4286 0008:HarmonicMean :2 3 6:3.0000
Each line represents a single test case. Each case has four fields separated by a ':' character delimiter (using a colon as a delimeter was an arbitrary choice). The first field is a test case ID, the second field signifies which method to test (I added spaces after GeometricMean and HarmonicMean purely for readability), the third field is a space-delimited list of inputs to the method, and the fourth field is the expected result. Placing test case data in a text file is simple and effective. Good alternative schemes for test case data storage are XML files, SQL databases, and Excel spreadsheets.
Where do the input values and expected results come from? As I mentioned in the previous section, examining the source code of the methods under test tells you how to call the methods, but it does not give you enough information to create a meaningful set of inputs and expected values for the test cases. A common practice—but a big mistake—is to use the system under test to generate expected results. In other words, you should not launch the application, enter some input integers, use the application to get a result, and then use the result as your test case expected result. This approach does not test the methods in the system under test; it just verifies that you get the same output given the same input!
So just how do you determine expected results? The theoretical guiding principle here is that you should use the product's specification documents. In theory, at least, the system under test will have documents that completely and precisely describe the product's behavior. Of course, the reality is that in a production environment the specification documents are often incomplete or out of date, and during the development cycle they may not have been written yet. The actual guiding principle for creating test cases is to do whatever is necessary: software testing is a practical endeavor. For the test cases in this example, to generate input and expected results I simply went to my old college statistics book and found some quizzes and solutions (you should always seek out some authoritative reference, like this statistics book, if at all possible). Additionally, you'll have to make use of software testing principles such as testing boundary conditions. Creating meaningful test cases is where experience and general knowledge of software testing comes into play.
The structure of the test automation in pseudocode is:
loop read a test-case line parse out test-case data build up input arguments call the method-to-test if actual value = expected value record a "pass" result else record a "fail" result end loop
There are several alternatives for test case storage and test harness structure, but this simple one has proven to be remarkably robust and flexible on many projects. An important test automation guideline is to keep everything simple. I begin by declaring the key variables that the automation uses:
Dim FS As New FileStream("..\TestCases.txt", FileMode.Open) Dim SR As New StreamReader(FS) Dim Line, CaseID, Method As String Dim Tokens(), TempInput() As String Dim Expected As String Dim Actual As Double
I open the test case's text file using classes from the System.IO namespace. The string variable line holds a single line representing one test case from the test case file. The variables caseID, method, and expected hold the test case ID, which method to test, and the expected result, respectively. Notice that the expected result is declared as a string type (because it is read in from a text file), but the actual result returned by the method under test is of type double so I'll have to do a type conversion at some point. This is an important rule of thumb in test automation: test case input arguments and expected values will usually initially be of type string (unless you use SQL for test case storage) so you'll have to convert the input to the appropriate data type. In addition, you'll also either convert the expected value from type string to match the actual value or convert the actual value to type string so that actual and expected values can be compared. Note that you should always explicitly convert data types and not rely on the implicit conversion available in some .NET-compliant languages like Visual Basic .NET. This means you, as the tester, are in control and have to think about the conversion process, adding another little push to get correct and accurate results while testing.
Figure 6** Processing a Test Case **
The string array variable tokens holds each part of a test case line. Variable tokens(0) holds the test case ID, tokens(1) holds which method to test, and so forth. String array tempInput holds each input argument. For example, if tokens(2) is "2 4 8", then tempInput(0) is "2", tempInput(1) is "4", and tempInput(2) is "8". Figure 6 shows the relationship between these variables for the first test case. Because these variables are logically related I could have placed them together into a struct or class but there are so few variables I wouldn't gain any clarity.
After displaying a simple output header line to the command shell, I'm ready to test. The test harness consists of a single test case data-controlled loop, as you can see in Figure 7.
Figure 7 Test Case Loop
Line = SR.ReadLine() While Line <> Nothing Tokens = Line.Split(":") CaseID = Tokens(0) Method = Tokens(1).Trim() TempInput = Tokens(2).Split(" ") Expected = Tokens(3) Dim Input(TempInput.Length - 1) As Integer For I As Integer = 0 To Input.Length - 1 Input(I) = Integer.Parse(TempInput(I)) Next Dim M As New MathLib.Methods If Method = "ArithmeticMean" Then ' test ArithmeticMean() ElseIf Method = "GeometricMean" Then ' test GeometricMean() ElseIf Method = "HarmonicMean" Then Console.WriteLine(CaseID & " " & Method & _ " Not yet implemented") Else Console.WriteLine("Unknown method") End If Line = SR.ReadLine() End While ' test case loop
After I read a line of test case data, I use the String.Split method to parse the line into the string array tokens. Then I use Split again to break the input arguments and store them into the string array tempInput. I could have used regular expressions but I find traditional string parsing techniques easier to debug. I branch on the value of the method to test, test the method, and log a pass or fail value based on the method's results.
Testing each method is relatively simple. But before I can call each method, I must convert the input arguments stored as string types in tempInput into integers:
Dim Input(TempInput.Length - 1) As Integer For I As Integer = 0 To Input.Length - 1 Input(I) = Integer.Parse(TempInput(I)) Next
I create a new integer array of the same size as tempInput and then use the int.Parse method to convert each string representation of an argument into its integer counterpart. Now I can test the ArithmeticMean method, like so:
Dim M As New MathLib.Methods Actual = M.ArithmeticMean(Input) If Actual.ToString("F4") = Expected Then Console.WriteLine(CaseID & " Pass " & Method & " actual = " _ & Actual.ToString("F4")) Else Console.WriteLine(CaseID & " *FAIL* " & Method & " actual = " _ & Actual.ToString("F4") & " expected = " & Expected) End If
After I get the actual result (as type double), I compare the actual result with the expected result to determine if the test case passed or failed. Notice that I converted the actual result from type double to type string formatted with four fixed decimal places. If I had converted the expected result from type string to type double then I'd have to be careful about comparing two double types for exact equality and worry about rounding because double values are sometimes only approximations.
Writing results to the command shell is simple and effective. Because the test harness is a console application, you can easily send results to a text file by using system redirection when you run the harness, as shown here:
C:\>TestAutomation.exe > results.txt
Testing GeometricMean is similar to testing ArithmeticMean:
Actual = M.GeometricMean(Input) If Actual.ToString("F4") = Expected Then Console.WriteLine(CaseID & " Pass " & Method & " actual = " _ & Actual.ToString("F4")) Else Console.WriteLine(CaseID & " *FAIL* " & Method & " actual = " _ & Actual.ToString("F4") & " expected = " & Expected) End If
Because GeometricMean is an instance method, I must instantiate an object of type MathLib.Methods and then call GeometricMean in the context of the object. After determining a pass or fail result, I write the results along with minimal details to the command shell. In an actual production environment you will want to add as much detail as possible to the output for failed test cases.
Discussion
You can extend and adapt the API test automation techniques I've presented here in several ways. Because this column is primarily instructional, I removed all error checking for clarity. In a production system you'll want to add exception handling try/catch-finally blocks to make sure your automation does not stop in mid-run. You'll also want to add counters that track the number of test cases that pass and fail and then log summary results. Another useful extension is to automate the automation—schedule the API test harness to run automatically (using the Windows Task Scheduler) and automatically send test run result summaries via e-mail (using the classes in System.Web.Mail namespace).
API test automation does not eliminate the need for manual testing, but automated testing has significant advantages over manual testing. First, automated testing can be much faster than manual testing—it is possible to run several hundred thousand test cases in the same time it would take to run only several hundred manual tests. However, you must also factor in the time required to design, write, debug, and maintain the automation suite. Often with automation, the initial investment is significant, and a substantial part of the benefit lies in the longer term payoff. Second, automated testing can be more accurate than manual testing—manual testing is subject to all kinds of human error. Third, automated testing is more precise in that automated tests are deterministic and run the same way every time given the same input. Fourth, automated testing is more efficient—automated tests can run anytime, freeing you to do other work. And fifth, automated testing actively promotes your skill growth while manual testing can be very repetitive.
The custom API test harness system I've described here nicely complements more structured third-party test frameworks like NUnit and even more sophisticated commercial test framework systems. In the days before .NET, writing custom test automation was so time consuming that it often wasn't done except on the largest projects. But the enormous productivity gains made possible by the .NET environment allow you to write custom automation quickly. The advantage of custom automation over frameworks written by someone else is that you can test interesting scenarios not anticipated by test frameworks. Also, when using a third-party test framework there is a risk that you'll test based on the capabilities of the framework rather than on the needs of your system under test. The disadvantage of writing custom test automation harnesses versus using preexisting test frameworks is the additional time you sometimes spend to create a custom harness and the possibility that you might introduce bugs into the harness.
There are four closely related ways you can use API test automation in a production environment. First, during the early stages of the product cycle, the development of API test automation itself often uncovers serious functional bugs in the methods used by the SUT. Second, during the middle stages of the product cycle, API test automation is useful as part of build verification testing—after you build a new version of the system under test, you can verify that the system meets some minimal threshold of functionality before moving forward with production. Third, during the latter stages of the product cycle you can use API test automation as part of developer regression testing—making sure that new code does not break existing functionality before you check the new code into your source control/build repository system. And fourth, before releasing the product you can use API test automation as part of a complete test pass.
With software systems growing in complexity and security becoming increasingly significant, thorough software testing is more important than ever before. API testing as I've described here is an absolutely essential part of your testing effort and the .NET environment makes it simple to implement automated API testing.
Send your questions and comments for James to testrun@microsoft.com.
James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.