Walkthru:Extending Test Types
|Extensibility Guided Tour|
|Walkthru:Extending Test Types|
Glossary Item Box
Extending Test Types
To support a growing reliance upon testing in the industry, Visual Studio 2005 Team System incorporates complex testing functionality directly within the Visual Studio IDE. Embedding testing functionality into Visual Studio allows developers easier access to load and unit testing without forcing a context switch to another tool. Furthermore, integrating testing into Visual Studio allows for metrics about testing to be easily marshaled against the Team Foundation Server, which helps improve communication. It is also possible for organizations and Visual Studio Integration Partners to extend the types of tests available by leveraging the Test Type Extensibility Framework to add their own fully integrated test types.
Test Types in Visual Studio Team System are implemented, at the lowest level, as VsPackages which enables them to integrate seamlessly with the Visual Studio IDE. Then, in order to interface with core testing components within the IDE, such as the Test Manager, Test View, and Test Results tool windows custom Test Types also need to implement the following, core, classes:
- TestType Class: Provides a GUID which can be used to permanently define the test type among the various components used to create the custom test type.
- TestElement Class: The core test functionality provided by the custom Test Type is implemented as a Test Element instance. Because this type may be marshaled to other machines in the process of testing, this class must be serializable.
- TestResult Class: Handles results for each test instance of the custom type, and stores them for interaction with reporting interfaces.
Because of the scope of extending Test Types, and the amount of code that you would be forced to copy and paste, this walkthrough will guide you through an existing example that ships with the VS SDK rather than walk you through the process of creating your own custom test type. By focusing on an existing example, you will be able to better get a feel for the type of extensibility possible in a short amount of time. Additionally, included in the VSTS section of the VS SDK, there is a whitepaper that provides extensive detail on what is needed to fully leverage the Test Type Extensibility Framework (which is too vast for the scope of this simple walkthrough).
1. Locate the VSTS Section of the VS SDK. The typical path to this location is:
C:\Program Files\Visual Studio 2005 DK\<build>\VisualStudioTeamSystemIntegration\
(Where <build> is the year and month of the build of the VS SDK that you currently have installed.)
2. Locate the Test Tool Extensibility folder, right click on it and select the Extract All menu option. Click Next to begin the Extraction Wizard. Click the Browse button, select the Desktop, and then click the Make New Folder button. Name the new folder "Extensibility Sample", click OK, and then click Next. Once extraction is complete, click Finish, and then from the Extensibility Sample folder, double click on MyTestSample.sln to open it in Visual Studio.
Note that you must have Visual C++ loaded or you will receive an error informing you that the UI package could not be loaded. If you get this error, you can continue with the walkthrough, but realize that you won't be able to build and deploy the sample application at the end of the walkthrough.
3. Once the MyTest project is loaded, expand the References node in Solution Explorer.
References to core assemblies required for Visual Studio Extensibility (such as the Microsoft.VisualStudio.Shell, and related assemblies) are listed, as well as references to core testing libraries. All of these libraries are required to successfully implement a VsPackage that can be used as a custom Test Type.
4. In Solution Explorer, double click on MyTestPackage.cs to open it in the editor.
Within the Test Type Extensibility Framework, VsPackages interact with Testing Components found in Visual Studio Team System through the use of special registration attributes. In this case, MyTestPackage exposes the RegisterTestTypeNoEditorAttribute (line 46 in the image above) to indicate to the testing framework that this package exposes a simple Test Type – one that doesn't provide an editor to allow testers to specify criteria and settings. The MyTestPackage VsPackage also exposes a Tool Window which is used to visually display test result information once instances of the MyTest test class are run. This information is provided to the Visual Studio IDE by way of the ProvideToolWindowAttribute (found on line 52 in the image above). (Both of these attributes, as well as other registration options, are detailed in the white paper accompanying the Test Extensibility section of the VS SDK.)
5. Note too that MyTestPackage.cs proffers a specialized Service to the Visual Studio IDE via the ProvideServiceForTestTypeAttribute (line 50 in the image above). Code to proffer the service is handled in the private OnCreateService() method (which gets mapped during class creation). In the OnCreateService() method, code merely ensures that needed input is in order, then creates and returns a new instance of the MyTestTuip class – which acts as a wrapper designed to house and administer various Test Type related functionality and interfaces.
6. In Solution Explorer, double click on MyTestTuip.cs to open it. Near the bottom of the file is a public interface used to identify the SMyTestService. This interface, in turn, is implemented by the MyTestTuip class (found near the top of the file). Together these objects inform the Visual Studio IDE of the presence of the MyTest Test Type during run time. In other words, these objects merely serve to facilitate communication between the custom Test Type and the IDE – they're not the implementation of the Test Type itself.
With the necessary framework now in place to expose testing functionality via the proper registration attributes, and to manage interaction with the IDE, the custom functionality desired for a new Test Type can be implemented. Providing custom test functionality in Visual Studio Team System requires the implementation of four key classes:
It is with these classes that base functionality for a new Test Type is provided. (Additional functionality such as custom editors, custom viewers, or the ability extend the run configuration settings require the implementation of other classes.)
7. In Solution Explorer, double click on MyTest.cs to open it in the editor. MyTest implements the TestElement class supplied by the Test Type Extensibility Framework, and serves to describe the properties of a test during run time. Because this class may end up being marshaled between machines during a distributed test, this class must be serializable - hence the SerializableAttribute above the class's definition.
8. Expand the TestType instance code region. Near the top of the MyTest class, note that the class exposes a static TestType class with a hard-coded Guid. This Guid, or TestType marker, will be used to uniquely identify the custom type (not instances of the type) and all of its constituent elements during run time.
9. Expand the Constructors code region. Now scroll down to the constructors for MyTest. The first constructor is the default construct required for serialization. The comments for the second instructor indicate that this constructor is usually called by the Test Tip (or TIP).
In the Testing Framework, tests are processed at run-time, but users frequently wish to save test results for future use (to detect trends, avoid regression, or analyze performance changes, etc.). The purpose of the TestTip class is to tips, or instructions, to the custom Test Type on how to serialize and load results. In other words, the TIP is where Test Type designers get to specify the serialization mechanism for their Test Types. Because they are responsible for the implementation details, they are free to chose a serialization format that makes sense to their particular test domain. A simple storage mechanism would obviously be to just persist test data to disk in a flat file, or maybe as xml. However, because the implementation details are left to the designer, it might make sense to store the data in a SQL Server database, or to expose a custom object that could hydrate and dehydrate data through the use of a Web Service. The core concept, of course, being flexibility – the flexibility to allow Visual Studio customers and Partners the ability to store and load test data for their own test extension types in whatever manner makes sense.
10. In Solution Explorer, double click on MyTestTip.cs to open it in the editor. Note first that it exposes a single, mandatory, property: TestType – where it returns the TestType used to uniquely identify the custom type. Look at the implementation details for the Load() and Save() methods. As you can see, these methods just persist data to a simple StreamWriter. Note, however, that they make use of collections of ITestElement objects as a handle for the serialization and deserialization process. It is here that the MyTest (which subclasses the TestElement base class) instances are created from disk, or saved to disk.
11. Close MyTestTip.cs, and return to MyTest.cs.
With a serialization and deserialization mechanism now in place, coupled with descriptive information about the nature of the test itself (and a framework to inform Visual Studio and the Test Type Extensibility Framework that the test type is available), there now just remains one final detail: how the test actually interfaces with Visual Studio (or end users) at run time to actually perform custom test logic.
12. Expand the TestElement overrides code region, then scroll down and find the Adapter property. Other properties in this class serve to describe the type of functionality exposed by this custom Test Type (such as whether or not the test is read only, or can be invoked from the command line, etc.). The Adapter property returns an object – which points to the MyTestAdapter class.
The function of the Adapter is to allow custom test logic to be tightly integrated into the Test Framework provided by VSTS. In other words, the VsPackage and supporting objects are used to register a Test Type with Visual Studio (to allow new instances of the Test Type to be loaded by end users, or testers), but the Adapter implementation actually handles the custom logic and testing functionality at run time – it's purpose is to bridge the gap between your testing objects (for serialization and reporting) and the existing framework which actually processes tests.
13. In Solution Explorer, double click on MyTestAdapter.cs to open it in the editor. Scroll down a bit, expand the IBaseAdapter methods code region, and locate the Run() method. In this method, an ITestElement (an instance of MyTest in this case) is passed in, along with an instance of a testing context object for processing. Results from each run are stored in a new instance of the MyTestResult class – which is created just above the try/catch block. Within the method itself, some simple code is wrapped in a timer block, and outcome as well as duration are then recorded by setting properties on the result instance, and by registering the result instance with the test context (near the end of the method).
14. If you wish to compile and run the sample, make sure to follow the instructions found in the readme.txt file that accompanies the solution. (And don't forget the batch files that accompany the Solution in the Extensibility Sample folder).
For more information on the nature and avialble functionality presented by the Test Type Extensibility Framework provided in Visual Studio Team System make sure to check out the whitepaper in the VSTS section of the VS SDK.