Building and Testing Devices with the Windows CE .NET Test Kit (CETK)
Download the cetk_lab.exe sample
Microsoft® Windows® CE .NET
Summary: The Windows CE .NET Test Kit (CETK) is a powerful tool that can greatly increase the reliability of a platform. The tool can be used on a platform currently in development or a platform on a retail device. This paper describes the usage of the Windows CE .NET Test Kit, Board Support Package (BSP) Certification, and Driver Validation. (12 printed pages)
The Microsoft® Windows® CE .NET Test Kit (CETK) is a tool that can be used to test individual device drivers or an entire platform. The test tools in the CETK provide feedback on the functionality of your drivers, which can in turn be used to further enhance the reliability of your device. Additional tests can be added to the Windows CE .NET Test Kit to test specific drivers.
The Windows CE .NET Test Kit (CETK) provides the Windows CE .NET community with a common testing platform on which to test embedded devices and drivers based on Windows CE .NET. The Windows CE .NET Test Kit ships with the same tests that are used within Microsoft to verify the reliability and stability of platforms and drivers. The Windows CE .NET Test Kit was created to provide OEMs with a quick and efficient method of testing drivers. The primary goals of the Windows CE .NET Test Kit are to:
- Create a tool that is quick and easy to use.
- Automate the test process as much as possible for OEMs so that testing is faster and more accurate.
- Create a tool that can be used for BSP and Driver Certification.
Who Should Use the CETK
The CETK can be used by anyone who wants to test and verify the stability and reliability of a device, BSP or specific drivers. The CETK tests ensure that a device or driver is meeting a minimum standard of stability and compatibility. An OEM who uses the CETK to test a driver or device is sure that that it is compatible with the Windows CE .NET operating system.
PC and Device-Side Hardware Requirements for Running CETK Tests
The Windows CE .NET Test Kit uses device-side (client) and desktop (server) applications, which exchange information over sockets. The client that runs on the Windows CE .NET device exchanges information with the host application, which typically resides on the development workstation running Windows CE .NET Platform Builder. Target devices need working Ethernet and display support to run the Windows CE .NET Test Kit.
How Using the CETK Can Improve Platform Reliability
The CETK takes into account years of testing experience on Windows CE .NET to bring the tests that are the most valuable, comprehensive, and easy to automate to the embedded community. Making the tests both simple to use and easy to understand has been very high on the priority list for the CETK. The CETK tests verify that a driver and device running the Windows CE .NET operating system is meeting a standard of reliability and stability. The standard for device compatibility is established with the Standard Development Board (SDB) specification. This specification was written to help OEMs build compatible hardware and software for the Windows CE .NET operating system. The Windows CE .NET Test Kit has tests that will help assure that devices and drivers are following this standard. Developers typically develop custom device drivers for new hardware designs. The CETK is designed to assist developers in building robust drivers, and therefore reliable devices. The designers are always working to expand the test kit to be more comprehensive in what it tests and how it works with both debug and shipping devices.
Similarities and Differences Between the CETK and Desktop WHQL Tests
The CETK is not a part of Windows Hardware Quality Labs (WHQL). Users that are familiar with WHQL can consider the CETK to be the WHQL program for Windows CE .NET. WHQL tests are developed to work with Windows desktop operating systems. The architecture and structure of the WHQL tests are not compatible with Windows CE .NET, which is why the Windows CE .NET Test Kit was created. Microsoft offers two programs that are somewhat similar to WHQL tests: the BSP Certification program and the Driver Verification program. Both of these programs use the CETK to test and validate drivers and BSPs.
Which Tests Are Supported Out of the Box
The current set of tests in CETK is a collection of the most common types or drivers and devices for Windows CE .NET. The CETK is constantly expanded to include more and a wider variety of tests. Each test is fully documented in what it tests and how to use it in the CETK Help.
Windows CE .NET Test Kit—Shipping Tests
- Bluetooth Test
- CD/DVD-ROM Block Driver Test
- CD/DVD-ROM File System Driver Test
- Direct3D Basic Rasterization Test
- Direct3D Fogging Test
- Direct3D Multi-Texturing Test
- Direct3D Z-Buffering Comparison Test
- DirectDraw Test
- DirectSound Audio Driver Test
- File System Driver Test
- Graphics Device Interface Test
- IR Port Test (Winsock 1.1)
- IR Port Test (Winsock 2.0)
- Keyboard Mapping Test
- Keyboard Test
- Mouse Test
- One-Card Network Card Miniport Driver Test
- Parallel Port Test
- Partition Driver Test
- PCMCIA Controller Driver Test
- Printer Driver Test
- Serial Communications Test
- Serial Port Driver Test
- Smart Card Resource Manager Test
- Storage Device Block Driver API Test
- Storage Device Block Driver Benchmark Test
- Storage Device Block Driver Read/Write Test
- Telephony API Client Test
- Telephony API Server Test
- Touch Panel Test
- Two-Card Network Card Miniport Driver Test
- Waveform Audio Driver Test
- Winsock 2.0 Test (v4/v6)
- Winsock Stress Test
The test kit can be used at any time during the development of a platform, or on shipping devices. The CETK can be used to test the stability of a system as a whole or it can be used on specific drivers under development.
Adding the CETK Components to a Platform
Before you use the CETK, you need to add support for the CETK to your platform. This can be achieved in one of two ways: either by adding support for the CETK to a platform by adding the CETK component from the Platform Builder catalog, or by copying files to the platform before running the CETK tests.
The CETK requires an Ethernet connection from the target device to the network on which the development workstation resides. You cannot run the CETK tests over a debug Ethernet connection. Note that you can run the CETK tests against a Windows CE .NET operating system that has been built as a debug image. In this case, you need a full network driver included in your platform, not a debug Ethernet driver.
Note To use the CETK, do not build your platform from the Tiny Kernel configuration. You cannot run the CETK on a platform built from the Tiny Kernel configuration.
To add the CETK tests to your platform in Platform Builder
- In the Catalog, expand the Device Drivers node, right-click Windows CE Test Kit, and then click Add to Platform.
- On the Build menu, click Build Platform to build the operating system image for your platform.
- After the Windows CE .NET operating system image has been built, download the operating system image to an appropriate Windows CE .NET-based device.
Note Including the CETK component from the catalog does not add any additional files to your operating system image. Some CETK required files will be placed in your build release folder, including the file clientside.exe.
Because the CETK components are not included in your operating system image, you must download and run the client side CETK application, called clientside.exe.
To run the CETK client software (clientside.exe) on your operating system image
- On the Platform Builder Target menu, click Run Programs.
- Click Clientside.exe, and then click Run.
Note that the Run Programs dialog box lists all the executable code found in your platform's build release folder. This list may include applications that are not part of the running operating system image [nk.bin]).
You can now run the CETK tool on your development workstation.
To run the CETK tool on your development workstation
- Click Start, point to All Programs, point to Microsoft Windows CE .NET, and then click Windows CE .NET Test Kit.
- To run the CETK tool without adding the CETK client software to your OS image, download the Clientside.exe file to your Windows CE .NET-based device.
On the development workstation, the Clientside.exe file is located in a directory with a name corresponding to the CPU for your target device, in the %_WINCEROOT%\Others\Wcetk directory.
- On your Windows CE .NET-based device, on the Start menu, click Run and then type the location of the Clientside.exe file followed by the name and port number of the CETK server.
For example, by typing clientside /n=machine1 /p=5555 you can run the Clientside.exe executable file and connect to machine1 using port number 5555. You can also edit the Wcetk.txt file located in the directory in which the Clientside.exe file is located. Platform Builder creates a Wcetk.txt file in the release directory for your platform if you add support for the CETK to the platform prior to building the platform.
You can also modify the following registry key on the target device: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CETT. Clientside.exe reads settings from this registry key when a command line and a Wcetk.txt file are not found. If you are building a custom operating system image using Windows CE .NET Platform Builder, you can modify the registry of the target device by editing the product.reg file for your platform, including the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CETT registry key. When you build the operating system image, the information from the Product.reg file is built into the operating system. As an alternative, you can use the Remote Registry Editor tool to modify the registry of a target device while the target device runs.
PC Side Software
The PC side or Host side portion of the CETK is where all the work is done. All the tests are stored on the PC side, and that is where devices are controlled. The Host Side user interface gives the user an environment for testing devices that is both simple and flexible. There are many features that you can use to help manage the testing of individual drivers or a complete device.
The Test Suite Editor dialog box allows you to create and manage test suites for the CETK. A test suite is a selection of tests that is stored by the CETK and can be applied to any target device in the CETK window.
You can use the Test Suite Editor to create a test suite that contains tests that you want to run. Rather than selecting each individual test on a target device, you can apply the test suite to the target device to automatically select the tests that you selected in the Test Suite Editor.
A user can also change the default command line for a test to change how that test runs and what is tested. Each test usually has many other options to add further functionality testing.
Device Side Software
The device side of CETK is called Clientside.exe. Clientside.exe is a small application that is responsible for calling to the host computer to establish a connection, device detection, launching tests, and sending back the results to the host.
Tux and Kato
Tux is a standard method of writing and executing test code on a Windows CE .NET device. The main component of the Tux model is the executable file tux.exe, which is responsible for launching and controlling the tux.dll file.
CETK provides users with all of the necessary files and instructions to build custom tests that can be added into the CETK by using the User Test Wizard, which can be accessed from the Host UI.
The Tux client, tux.exe, is the main component of Tux. It can be run in stand-alone mode or in conjunction with the CETK. The Tux client is a console application that uses the following syntax:
tux [-b] [-e] [-s file_name | -d test_dll] [-c parameters] [-r seed] [-x test_case] [-l] [-t address] [-h] [-k address] [-o] [-f file_name] [-a]
The following table shows the various Tux parameters.
|-b||Break after each Tux DLL is loaded.|
|-e||Disables exception handling.|
|-s file_name||Specifies the Tux suite file to load/execute.|
|-d test_dll||Specifies the Tux test DLL to load/execute.|
|-c parameters||Command line to pass to Tux DLL.|
|-r seed||Specifies the starting random seed.|
|-x test_case||Specifies which test case(s) to run.|
|-l||List of all the test cases in the Tux DLLs.|
|-t address||Specifies the computer running Tux server. Use -t with no arguments to specify a local server.|
|-h||Generates the Tux parameter list.|
Tux can also use a logging .dll file called Kato, which allows for a common logging interface. The Kato Logging Engine (Kato) is 32-bit, object-oriented, client/server logging engine that allows applications to log to a single interface and have their output routed to multiple user-defined output devices. Kato is completely thread-safe and process-safe. Kato provides an ANSI and UNICODE interface for a Windows–based desktop operating system.
The CETK will automatically use Kato as the method of logging so that the results from all of the CETK tests have similar results that make them easier to understand.
Interpreting the CETK Tests Results
To view the results of any test that has run, right-click the test and then click View Results. This will open the latest results for the selected test. The first information displayed is the summary of the test. Each test is actually a suite of tests; it is possible for some tests within each suite to pass and others to fail. If any test within the suite fails, the CETK will fail the entire suite of tests. For example, if you run the Winsock test on a device with only IPV4, the test will display as a fail and if you view the results you will see 2 passes and 2 fails. The 2 fails come from the fact that the device does not contain IPV6, so in this case the fails are expected. To have the Winsock test pass completely, you need to have IPV4 and IPV6.
Test Failures and How to Interpret
Because each test in the CETK is a suite of tests, you can have some tests within the suite fail and some that pass. Figure 1 shows the Winsock test results. There are four tests that are part of the Winsock test suite. In this scenario, two tests passed and two failed. If you select the test that failed, the window below the test list will show the results of that specific test. The results explain what was being tested and what was expected to be returned. The sources to the tests are also provided to help you further investigate failures. The sources are located on the last CD of Platform builder in a folder called CETK_Sources.
Figure 1. Winsock test results
The Board Support Package (BSP) Certification program was established to create standards of reliability and stability to third-party devices and software that are designed for Windows CE .NET.
The Microsoft Windows CE .NET BSP and CPU Certification Process is designed to certify Windows CE .NET BSPs and CPU compatibility. Certification by Microsoft carries the following benefits:
- It guarantees that the BSP and CPU meet the requirements set by the Standard Development Board Specification of the Microsoft Windows CE .NET operating system, and therefore will be more readily accepted by customers and OEMs.
- When the BSP and CPU pass certification testing, they will be added to the Supported BSP and Supported Processor lists.
- Inclusion on this list is a marketable quality assurance point and provides positive industry exposure.
The goals in creating the program were to make it simple but efficient. The CETK provides all the tests that used for certification. This makes it easy for OEMs and device manufacturers to test the device or BSP before sending it to Microsoft for certification. For further information about the program, please visit BSP and CPU Certification Process for Windows CE .NET.
How to Submit Hardware for Certification
When you are ready to submit your hardware for certification, you need to fill out the request form for the version of Windows CE that your device is running. The forms are at the BSP Certification Web site.
Please submit the following to the address listed below:
- Two identical devices.
- An operating system image, ready to be loaded on the device with the BSP that will be certified.
- A list of all applicable CETK tests. This is your completed BSP and CPU Certification Request Form.
- Test fee: A testing fee of $1,500 per BSP will be collected at the time of submission. This fee pays for up to two test passes. Should you need to retest, an additional fee of $1,000 will apply.
Note Include either a purchase order number in the appropriate field on the form or a check. Please make your check payable to Microsoft Corporation and include the check with the boards and the completed BSP and CPU certification request form at the time of submission.
One Microsoft Way
Redmond WA, 98052
ATTN: James Zwygart
If you have further questions or comments on the BSP/CPU Certification program, please send an e-mail message to email@example.com.
What Tests to Run Before Applying for Certification
The tests that will be used for Certification depend on the device you are submitting. Because each device might be configured differently, the CETK does most of the work for you by means of auto-detecting which peripherals are on the device. It is still very important to verify that all the applicable tests for your hardware are selected.
How to Interpret the Results from a BSP Certification Test
The BSP Certification program uses the CETK as the method of testing so you interpret the results in the same way as you would any CETK test pass or fail.
How Reliability and Certification Go Hand in Hand
Certification of your device or BSP provides assurance that your device meets the Microsoft standard for stability and reliability. To understand the baselines set by Microsoft, visit the Standard Development Board Specification. The Windows CE Test Kit follows the Standard Development Board specification, and therefore verifies that your Board Support Package meets the baseline requirements for a Standard Development Board.
The purpose of the Driver Validation Program (DVP) is to provide developers, Independent Hardware Vendors (IHVs) or OEM licensees of Microsoft Windows CE .NET the opportunity to validate their drivers for use with Windows CE .NET operating systems.
This validation program offers the following benefits:
- It confirms that, based on the test results submitted, the driver is compatible with the designated Windows CE .NET operating system at the time of submission.
- When the driver passes validation testing, it will be added to the list of Windows CE .NET Supported Device Drivers.
Inclusion on this list indicates that the validated device driver met Microsoft standards for compatibility with Windows CE .NET and provides positive industry exposure.
To submit drivers for validation
The submission process is very simple and easy. The Driver Validation Program Web site outlines the process and provides the forms for submission.
- Test your driver. You must test your driver by running the relevant tests from the CETK included with Platform Builder 4.0 for Windows CE .NET. Such tests must be run in accordance with the instructions and guidelines that accompany the CETK.
- Fill out the submission form. Download the driver validation submission form and complete it.
If you want to submit your driver to the Microsoft corporation for hosting on a Microsoft Web site and distribution to end users, you must submit your device driver as an .msi file created by using Windows CE .NET Platform Builder. You must also create a .cec file for the device driver in the MSI package. Together, the .MSI and .cec files allow for easy integration of the device driver into the Platform Builder IDE catalog with a one-click operation by end users. For more information about the tools provided in Platform Builder that help you create .cec and .msi files for your device driver, please refer to Device Driver Integration into the IDE in the Platform Builder documentation.
Note Microsoft is not responsible for determining end-user licensing terms for your driver. It is your responsibility to include with your driver files, either a README file or an End User License Agreement (EULA) regarding any conditions on the use of your driver. The driver is your responsibility. Microsoft has no responsibility for any technical or customer support, updates, marketing or any other aspect of the driver.
- Send in the form. Send an e-mail message to Windows CE Driver Validation Program Submissions and attach your submission form, driver, and CETK logs.
What Role Does Driver Validation Play for Reliability of a System
Validation of your device driver provides assurance that your device driver meets the Microsoft standard for stability and reliability. To understand the baselines set by Microsoft, you are encouraged to read the device driver documentation that ships with Windows CE .NET.
Microsoft is dedicated to providing the tools and documentation to assist developers in building reliable devices and device drivers. Microsoft provides the BSP certification program and the driver validation program, which can be used by OEMs to verify that a device meets the Microsoft standards for stability and reliability.
Microsoft is interested in obtaining feedback on the current generation of BSP and driver testing tools. To provide feedback, send an e-mail message to the following e-mail addresses:
For the latest information on Windows CE .NET, please visit the Windows Embedded Web site.
The online documentation and context-sensitive Help included with Windows CE .NET also provide comprehensive background information and instructions for using Windows CE .NET.
You can also visit the product documentation for Windows CE .NET on the Windows Embedded Web site.