Windows Phone Marketplace Test Kit
April 17, 2012
The Windows Phone Marketplace Test Kit provides a suite of automated, monitored, and manual tests to help prepare your applications to be accepted in the Marketplace the first time you submit them. The test kit enables you to identify and fix issues prior to Marketplace submission, which saves time in the submission process. You can use the test kit to evaluate applications that target Windows Phone OS 7.1 or later.
The Marketplace Test Kit is a component of the Windows Phone SDK. For information about how to install the Windows Phone SDK, see Installing the Windows Phone SDK.
Running the test kit on your application will help you determine whether or not the application will pass Marketplace certification. If a test fails, the test kit provides details about what you need to fix in the application prior to submitting it for Marketplace evaluation. The following lists some of the tests in the test kit.
-
Whether the specified images and screenshots meet certification requirements.
-
Whether the application icon and background image used in the application meet the certification requirements.
-
What capabilities the application uses.
-
Whether the XAP meets size requirements and whether the application manifest is valid
-
Whether the application handles errors properly.
-
Whether the application starts quickly enough to meet certification requirements.
-
Whether the application exceeds recommended memory usage requirements. This is particularly important when developing for 256-MB devices.
-
Whether the Back button in the application is being used properly.
-
Whether the application uses APIs that are not allowed on the phone.
-
Whether the application uses APIs that are not allowed with background agents.
-
Whether there are unhandled exceptions in your application and whether the application closes properly.
The test kit is integrated within Visual Studio. Use the following steps to open the test kit.
-
In Visual Studio, open a Windows Phone application solution that targets Windows Phone OS 7.1 or later.
-
In Solution Explorer, select the project you want to test.
-
On the Project menu, choose the Open Marketplace Test Kit option.
-or-
Right-click the project to show its context menu and select the Open Marketplace Test Kit option.
The test kit opens in a new Visual Studio tab named Marketplace Test Kit. The following image shows the test kit open in Visual Studio.
At the bottom of the test kit tab, you may see a yellow bar that says the test cases have been updated. The following image shows the update prompt.
-
If you see an update prompt at the bottom of the test kit tab, click the Update button. If you update the test kit, you should close and restart it prior to proceeding.
Since the test kit opens in a new Visual Studio tab, you can easily switch between files in your application and the test kit. Within the test kit interface, there are tabs that list the tests. These tabs enable you to switch between the features of the test kit. Each tab contains a different set of tests to run on the application.
Note:
|
|---|
|
All tests will be run on the project that was selected when the test kit was opened. You can run multiple instances of the test kit that target different projects at the same time. If you want to open a test kit for another project, select the project in Solution Explorer and select Open Marketplace Test Kit from its context menu. Otherwise, to change the target project, you must close the test kit, select the project, and then reopen the test kit. |
The test kit has multiple tabs that contain application details and test categories. Each test category lists a series of tests with a name and description. The tests are categorized by whether they can be run automatically or whether they need partial or full participation from a developer. The test kit contains the follow tabs:
-
Application Details
-
Automated Tests
-
Monitored Tests
-
Manual Tests
The following sections discuss these tabs and how to run the tests in more detail.
Application Details
The Applications Details tab lists the application package and enables you to specify images that will be evaluated later in the testing procedure. The following image shows the Application Details tab in the test kit.
The Application package box lists the path to the application package (XAP file) for your application, which by default is the Bin\Release folder of the currently selected project. This means you must use a release build to pass the automated and monitored tests.
The Applications Details tab also enables you to specify the images that will represent your application in the Marketplace. These images will be tested as part of the test kit. If you do not specify the required images, your application will not pass the tests in the test kit. The following table lists the images that you can specify.
|
Note:
|
|---|
|
Be aware that the image requirements can change. Be sure to test your application with an updated version of the test kit, and check the certification requirements on MSDN. For more information about image requirements, see Application Submission Requirements. |
|
Image |
Size (pixels) |
Format |
Required |
|---|---|---|---|
|
Large application tile |
173 x 173 |
PNG |
Yes |
|
Small application tile |
99 x 99 |
PNG |
Yes |
|
|
200 x 200 |
PNG |
Yes |
|
Application screenshot |
480 x 800 |
PNG |
Yes |
|
Up to 7 additional application screenshots |
480 x 800 |
PNG |
No |
For more information about the artwork required to submit an application, see Application Artwork.
You can use the screenshot tool provided by the emulator to easily obtain screenshots of your application that are the correct size and resolution. For more information, see How to: Create Screenshots for Windows Phone Marketplace.
Automated Tests
The Automated Tests tab provides tests that evaluate the basic criteria of your application, such as application size, capabilities required by your application, and the presence and size of application images. The following image shows the Automated Test tab in the test kit.
|
Note:
|
|---|
|
Be aware that certification requirements can change. Be sure to test your application with an updated version of the test kit, and check the certification requirements on MSDN. For more information about the certification requirements for the automated tests, see Application Submission Requirements. |
To run the automated tests
-
In the Marketplace Test Kit, on the Application Details tab, make sure the additional resources for your application are specified.
-
Create a release build. For more information, see How to: Build and Deploy a Windows Phone Application Using Visual Studio.
You must use a release build to pass the automated tests.
-
Click the Automated Tests tab.
The automated tests display.
-
Click the Run Tests button.
The automated tests start. When the tests complete, you will receive a pass or fail result for each test. You will also receive a description of the test and any error messages that are relevant.
-
If necessary, use the results of the Capability Validation test to update the WMAppManifest.xml file with the correct information.
For more information about how to do this, see How to: Determine Application Capabilities.
-
If a test fails, use the provided information to fix the issue and run the tests again. You should repeat this process until your application passes all the tests in this section.
Monitored Tests
The Monitored Tests tab provides a suite of tests that analyze the performance and reliability of your application during use on a device. The following image shows the Monitored Tests tab in the test kit.
|
Note:
|
|---|
|
Be aware that the performance and reliability requirements can change. Be sure to test your application with an updated version of the test kit, and check the certification requirements on MSDN. For more information about the requirements for the monitored tests, see Technical Certification Requirements. |
To run the monitored tests
-
In Visual Studio, create a release build. For more information, see How to: Build and Deploy a Windows Phone Application Using Visual Studio.
You must use a release build to pass the monitored tests.
-
On the Standard toolbar, set the target device to Windows Phone Device.
Performance on the emulator does not represent the actual performance. You must run on a device to obtain accurate measurements.
-
Connect a registered Windows Phone device to your computer. Make sure the device is powered on and the screen is unlocked.
-
In the Marketplace Test Kit, click the Monitored Tests tab.
The monitored tests will display.
-
Click the Start Application button.
This deploys your application to the tethered device, starts the application, and then starts the monitored tests.
-
Navigate through your application and perform common user tasks. In addition, you should use the Back button on the device while your application is running.
-
When you have completed navigating through the application, close the application. You can close the application by clicking the Back button or by clicking the Close Application button on the Monitored Tests tab. Clicking the Back button is preferred method of closing.
After the application is closed, the test results display. For each test, the Result Details column will be updated with Passed or Failed to indicate the status of the test or Not Analyzed if there was insufficient data to run the test. If the test was run, more details about the test results display in the Error Message column.
-
If a test fails, use the provided information to fix the issue and run the tests in this section again. You should repeat this process until your application passes all the tests in this section.
Manual Tests
The Manual Tests tab provides a series of test cases that you should complete with your application running on a Window Phone device. The manual tests require you to navigate through your application and observe its behavior in several different conditions to make sure it meets the application certification requirements. The following image shows the Manual Tests tab in the test kit.
|
Note:
|
|---|
|
Be aware that the certification requirements can change. Be sure to test your application with an updated version of the test kit, and check the certification requirements on MSDN. For more information about the certification requirements for the manual tests, see Technical Certification Requirements. |
To run the manual tests
-
In Visual Studio, create a release build. For optimal results you should also specify Windows Phone Device as the deployment target. For more information, see How to: Build and Deploy a Windows Phone Application Using Visual Studio.
-
On the Standard toolbar, set the target device. For optimal results you should specify Windows Phone Device.
-
In the Marketplace Test Kit, click the Manual Tests tab.
The manual tests display.
-
For each test, follow the steps provided in the Test Description column.
-
Observe your application’s behavior and change the drop-down in the Result column to Passed or Failed depending on the results you receive.
-
If a test fails, use the provided information to fix the issue. You can get more information about the certification requirement for the test by clicking the More info… link in the Test Description column.
-
When you have fixed the issues with your application, redeploy it to a device and walk through the test steps again. You should repeat this process until your application passes all the tests in this section.
