1 out of 5 rated this helpful - Rate this topic

Windows App Certification Kit tests

[This topic is featured in Develop great apps for Windows 8.]

The Windows App Certification Kit contains a number of tests that can help ensure that your app is ready to be published on the Windows Store.

Tests for Windows Store app certification

Tests for Desktop app certification

Tests for Windows Store app certification

Crashes and hangs test

Monitors the app during certification testing to record when it crashes or hangs.

Background

Apps that stop responding or crash can cause the user to lose data and have a poor experience.

We expect apps to be fully functional without the use of Windows compatibility modes, AppHelp messages, or compatibility fixes.

Apps must not list DLLs to load in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs registry key.

Test details

We test the app resilience and stability throughout the certification testing.

The Windows App Certification Kit calls IApplicationActivationManager::ActivateApplication to launch Windows Store apps. For ActivateApplication to launch an app, User Account Control (UAC) must be enabled and the screen resolution must be at least 1024 x 768 or 768 x 1024. If either condition is not met, your app will fail this test.

Corrective actions

Make sure UAC is enabled on the test computer.

Make sure you are running the test on a computer with large enough screen.

If your app fails to launch and your test platform satisfies the prerequisites of ActivateApplication, you can troubleshoot the problem by reviewing the activation event log. To find these entries in the event log:

  1. Open eventvwr.exe and navigate to the Application and Services Log\Microsoft\Windows\Immersive-Shell folder.
  2. Filter the view to show Event Ids: 5900-6000.
  3. Review the log entries for info that might explain why the app didn't launch.

Troubleshoot the file with the problem, identify and fix the problem. Rebuild and re-test the app.

Additional resources

Windows Vista and Windows 7 Operating Systems

Using Application Verifier Within Your Software Development Lifecycle

Application Verifier

Using Application Verifier

AppInit DLLs

Minimize startup time (Windows Store apps using C#/VB/C++ and XAML)

App manifest compliance test

Test the contents of app manifest to make sure its contents are correct.

Background

Apps must have a correctly formatted app manifest.

Test details

Examines the app manifest to verify the contents are correct as described in the Package metadata requirements.

The app can declare the file extensions that it needs to associated with. It is possible for an app to declare an unreasonably large number of file extensions to associate with and create a bad user experience. This test checks to ensure that this isn't the case with the app being tested.

This test also ensures that Windows Store apps take dependencies on appropriate framework versions. If there is an inappropriate dependency, this test will fail.

The following is the matrix of frameworks and their applicability to Windows App Model versions.

FrameworkOS MinRuns on Windows 8Runs on Windows 8.1 PreviewSupports Windows 8 appsSupports Windows 8.1 Preview apps
Microsoft.Media.PlayReadyClient6.2.1YesYesYesYes
Microsoft.VCLibs.110.006.2.1YesYesYesNo
Microsoft.VCLibs.120.00.Preview6.3.0NoYesNoYes
Microsoft.WinJS.1.06.2.1YesYesYesNo
Microsoft.WinJS.2.0.Preview6.3.0NoYesNoYes

 

Corrective action

Review the app's manifest against the requirements described in the Package metadata requirements.

Windows Security Features tests

Binary Analyzer Test

Background

If the app doesn't use available Windows security methods, it can result in the increased vulnerability of a user's device.

Test Details

This test will verify if the app is using the appropriate Windows security methods.

Corrective action

Apply the required linker options (SAFESEH, DYNAMICBASE, NXCOMPAT, and APPCONTAINER) when you link the app. See Fixing Binary Analyzer Errors for more info.

Banned File Analyzer test

Note  This test is only available in the latest version of the Windows App Certification Kit provided with the Windows 8.1 Preview SDK.

Background

If the app includes files with known security issues, it can result in the increased vulnerability of a user's device.

Test Details

This test will verify that Windows Store apps are not taking dependencies on binaries that have known issues or security flaws. It will scan files set to install from the package and flag any binaries that are outdated.

Corrective action

Update or remove any references and usage of the files identified in the test results. See Fixing banned file errors for more info.

Private Code Signing

Background

Including private code signing keys in the app package violates the Windows Store certification requirements.

Test Details

This test checks if the app package includes private code signing keys (for example, .pfx, .snk files).

Corrective action

Update or remove any references and usage of the signing keys identified in the test results.

Supported Windows Store API test

Test the app for the use of any non-compliant APIs.

Background

Apps must use the APIs for Windows Store apps (Windows Runtime or supported Win32 APIs) to be certified for the Windows Store. This test also identifies situations where a managed binary takes a dependency on a function outside of the approved Windows 8 or Windows 8.1 Preview profile.

Test details

  • Verifies that each binary within the app package doens't have a dependency on a Win32 API that is not supported for Windows Store app development by checking the import address table of the binary.
  • Verifies that each managed binary within the app package doesn't have a dependency on a function outside of the approved Windows 8 or Windows 8.1 Preview profile.

Corrective actions

Make sure that the app was compiled as a release build and not a debug build.

Note  The debug build of an app will fail this test even if the app uses only APIs for Windows Store apps.

Review the error messages to identify the API the app uses that is not an API for Windows Store apps.

Note  C++ apps that are built in a debug configuration will fail this test even if the configuration only uses APIs from the Windows SDK for Windows Store apps. See, Alternatives to Windows APIs in Windows Store apps for more info.

Performance tests

The app must respond quickly to user interaction and system commands in order to present a fast and fluid user experience.

The characteristics of the computer on which the test is performed can influence the test results. The performance test thresholds for app certification are set such that low-power computers meet the customer’s expectation of a fast and fluid experience. To determine your app’s performance, we recommend that you test on a low-power computer, such as an Intel Atom processor-based computer with a screen resolution of 1366x768 (or higher) and a rotational hard drive (as opposed to a solid-state hard drive).

Bytecode generation

As a performance optimization to accelerate JavaScript execution time, JavaScript files ending in the .js extension generate bytecode when the app is deployed. This significantly improves startup and ongoing execution times for JavaScript operations.

Test Details

Checks the app deployment to verify that all .js files have been converted to bytecode.

Corrective Action

If this test fails, consider the following when addressing the issue:

  • Verify that event logging is enabled.
  • Verify that all JavaScript files are syntactically valid.
  • Confirm that all previous versions of the app are uninstalled.
  • Exclude identified files from the app package.

Optimized binding references

When using bindings, WinJS.Binding.optimizeBindingReferences should be set to true in order to optimize memory usage.

Launch and Suspend performance

The performance of an app takes depends on many variables, such as system load and configuration, which can vary from one computer to another. Because of this variability, a launch or suspend time that is just under the limit in one instance, could be just over the limit, and fail the test, in another. If your app’s launch time or suspend time is close to the limit, review what the app is doing at these times and look for ways to reduce that activity.

Application suspend time is important for creating a fast and fluid experience for the user.

Test Details

Tests how quickly an app launches when the user starts it, how quickly it suspends when the user switches to another app, the peak memory usage, and the working set that the app uses.

Corrective Action

You should ensure that the app performance is consistent across different device configurations and does not exceed minimum requirements in order to avoid potential failure during Windows Store on-boarding. See Performance Best Practices for more info.

As low-power devices continue to evolve, their performance characteristics will change over time. Test your app with the most current version of the Windows App Certification Kit to ensure that your app complies with the latest performance requirements.

For general advice on launching and suspending apps written in JavaScript, see Optimizing your app's lifecycle and Performance best practices for Windows Store apps using JavaScript for an overview of best practices. If you are having difficulty meeting the load time limit and are writing in JavaScript, see Reducing your app's loading time for additional advice on how to build your app. C# developers can check out similar info in Minimize startup time.

When you run the Windows App Certification Kit, make sure that nothing else is running on the machine, as this will affect the results of the performance test. We don't recommend running the test on a remote desktop or in a VM, as these will also have a negative impact on the performance of your test.

This table has guidance for resolving Windows App Certification Kit errors from the launch and suspend test.

ErrorsDescription
Application Launch was not detectedClick on your app tile from the Start screen and make sure that the app launches correctly. If you are running tests on a remote machine (not a recommended practice), make sure that the terminal session window is not minimized.
Application Suspend was not detectedMake sure that your app suspends correctly. To check for Suspend, go to Task manager > View > Status values > Show suspended status. Launch your app and then switch to the desktop. If the app status changes to Suspended in Task Manger then the app suspended. If the app dies after few seconds, then your app fails to suspend correctly.
The Native Image Generator failedReview Troubleshooting Automatic Native Image Generation.
Infrastructure errorMake sure that your Windows App Certification Kit was successfully installed. If you are unsure that it was installed, see How to test your app.

 

Frequently asked questions

What are the performance limits for the launch test?

Here are the limits for the launch test:

x86/x64ARMNike
Pass< 3 seconds< 5 seconds< 6 sec
Warn3-5 seconds
Fail> 5 seconds> 5 seconds>=6 sec

 

Why do I get a warning when my app is launching in under 5 seconds?

Computers using an x86 or x64 processor are more powerful that low-power computers. An app that launches in 3 seconds on these computers may exceed the 5 second limit on a low-power system. The warning informs you that your app is at risk of failing windows store certification. If possible, test on a low-power computer.

What are the performance limits for the suspend test?

All computers have the same limits:

  • PASS: less than or equal to 2 seconds
  • FAIL: greater than 2 seconds

How do I measure the peak memory working set

The Task Manager provides a very convenient way to see the working set of your app during execution. Before you run the app, press Crtl+Alt+Delete and open the Task Manager. Select More details and open the processes tab. In the Options tab select Always on top so the Task Manager remains visible. The working set is shown in the Memory column.

To pass the Windows Store Performance test, your app must not exceed a working set of 500MB. To determine if your app meets this requirement, open it and run it through different scenarios to see how the app uses memory. As you run through the scenarios, keep an eye on the Memory column in the Task Manager. As the memory needs of your app change, the value in that column goes up or down. Keep in mind that if the Task Manager window is selected your app will lose focus which may affect the test. You can also use the Windows Performance Analysis tool to learn more about your app's memory usage and other performance issues.

How is performance measured?

Launch and suspend durations are calculated from the ETW events generated for the app under test. The time is calculated from the time difference between the Start and Stop event pairs of ETW events from the Microsoft-Windows-Immersive-Shell provider identified below.

Launch Test:

  • "PerfTrack_SplashScreen_AppShown/win:Start"
  • "PerfTrack_SplashScreen_AppShown/win:Stop"

Suspend Test:

  • "PerfTrack_PLM_SuspendPackage/win:Start"
  • "PerfTrack_PLM_SuspendPackage/win:Stop"

App manifest resources test

App resources validation

The app might not install if the strings or images declared in your app’s manifest are incorrect. If the app does install with these errors, your app’s logo or other images used by your app might not display correctly.

Test Details

Inspects the resources defined in the app manifest to make sure they are present and valid.

Corrective Action

Use the following table as guidance.

Error messageComments

The image {imageName} defines both Scale and TargetSize qualifiers; you can define only one qualifier at a time.

You can customize images for different resolutions.

In the actual message, {image name} contains the name of the image with the error.

Make sure that each image defines either Scale or TargetSize as the qualifier. For more info, see How to name resources using qualifiers.

The image {imageName} failed the size restrictions.

Ensure that all the app images adhere to the proper size restrictions.

In the actual message, {image name} contains the name of the image with the error.

For more info about your app's images and their sizes, see App images.

The image {imageName} is missing from the package.

A required image is missing.

In the actual message, {image name} contains the name of the image that is missing.

For more info about your app's images and their sizes, see App images.

The image {imageName} is not a valid image file.

Ensure that all the app images adhere to the proper file format type restrictions.

In the actual message, {image name} contains the name of the image that is not valid.

For more info about your app's images and their sizes, see App images.

The image “BadgeLogo” has an ABGR value {value} at position (x, y) that is not valid. The pixel must be white (##FFFFFF) or transparent (00######)

The badge logo is an image that appears next to the badge notification to identify the app on the lock screen. This image must be monochromatic (it can contain only white and transparent pixels).

In the actual message, {value} contains the color value in the image that is not valid.

For more info about your app's images and their sizes, see App images.

The image “BadgeLogo” has an ABGR value {value} at position (x, y) that is not valid for a high-contrast white image. The pixel must be (##2A2A2A) or darker, or transparent (00######).

The badge logo is an image that appears next to the badge notification to identify the app on the lock screen. Because the badge logo appears on a white background when in high-contrast white, it must be a dark version of the normal badge logo. In high-contrast white, the badge logo can only contain pixels that are darker than (##2A2A2A) or transparent.

In the actual message, {value} contains the color value in the image that is not valid.

For more info about your app's images and their sizes, see App images.

The image must define at least one variant without a TargetSize qualifier. It must define a Scale qualifier or leave Scale and TargetSize unspecified, which defaults to Scale-100.

For more info, see Guidelines for scaling to pixel density and How to name resources using qualifiers.

The package is missing a "resources.pri" file.

If you have localizable content in your app manifest, make sure that your app's package includes a valid resources.pri file.

The "resources.pri" file must contain a resource map with a name that matches the package name {packageFullName}

You can get this error if the manifest changed and the name of the resource map in resources.pri no longer matches the package name in the manifest.

In the actual message, {packageFullName} contains the package name that resources.pri must contain.

To fix this, you need to rebuild resources.pri and the easiest way to do that is by rebuilding the app's package.

The "resources.pri" file must not have AutoMerge enabled.

MakePRI.exe supports an option called AutoMerge. The default value of AutoMerge is off. When enabled, AutoMerge merges an app's language pack resources into a single resources.pri at runtime. We don't recommend this for Windows Store apps that you intend to distribute through the Windows Store. The resources.pri of a Windows Store app that is distributed through the Windows Store must be in the root of the app's package and contain all the language references that the app supports.

The string {string} failed the max length restriction of {number} characters.

Refer to the Package Metadata Requirements.

In the actual message, {string} is replaced by the string with the error and {number} contains the maximum length.

The string {string} must not have leading/trailing whitespace.

The schema for the elements in the app manifest don't allow leading or trailing white space characters.

In the actual message, {string} is replaced by the string with the error.

Make sure that none of the localized values of the manifest fields in resources.pri have leading or trailing white space characters.

The string must be non-empty (greater than zero in length)

For more info, see Package Metadata Requirements.

There is no default resource specified in the "resources.pri" file.

For more info, see Defining app resources.

There is no resource value specified in the “resources.pri" file.

Make sure that the app manifest has valid resources defined in resources.pri.

For more info, see Localizing the package manifest.

 

See App package resource validation for more info.

Branding validation

Windows Store apps are expected to be complete and fully functional. Apps using the default images (from templates or SDK samples) present a poor user experience and cannot be easily identified in the store catalog.

Test Details

The test will validate if the images used by the app are not default images either from SDK samples or from Microsoft Visual Studio.

Corrective actions

Replace default images with something more distinct and representative of your app.

Debug configuration test

Test the app to make sure it is not a debug build.

Background

To be certified for the Windows Store, apps must not be compiled for debug and they must not reference debug versions of an executable file. In addition, you must build your code as optimized for your app to pass this test.

Test details

Test the app to make sure it is not a debug build and is not linked to any debug frameworks.

Corrective actions

  • Build the app as a release build before you submit it to the Windows Store.
  • Make sure that you have the correct version of .NET framework installed.
  • Make sure the app isn't linking to debug versions of a framework and that it is building with a release version. If this app contains .NET components, make sure that you have installed the correct version of the .NET framework.

File encoding test

Background

HTML, CSS, and JavaScript files must be encoded in UTF-8 form with a corresponding byte-order mark (BOM) to benefit from bytecode caching and avoid certain runtime error conditions.

Test details

Test the contents of app packages to make sure that they use the correct file encoding.

Corrective Action

Open the affected file and select Save As from the File menu in Visual Studio. Select the drop-down control next to the Save button and select Save with Encoding. From the Advanced save options dialog, choose the Unicode (UTF-8 with signature) option and click OK.

Exceptions and waivers

Note  Applies to desktop app certification only.

{Exceptions and Waivers}

Direct3d feature level test

Direct3D feature level support

Tests Microsoft Direct3D apps to ensure that they work on all Windows 8 graphics hardware.

Background

Windows Store requires all applications using Direct3D to render properly on feature level 9_1. See Certification requirements for Windows apps for more info.

Test Details

The test will validate if the apps render accurately on feature level 9_1.

Corrective Action

Ensure that your app renders correctly on Direct3D feature level 9_1, even if you expect it to run at a higher feature level. See Developing for different Direct3D feature levels for more info.

Direct3D Trim after suspend

Note  This test is currently only available in the latest version of the Windows App Certification Kit provided with the Windows 8.1 Preview SDK, and only applies to Windows Store apps developed for Windows 8.1 Preview.

Background

If the app does not call Trim on its Direct3D device, the app will not release memory allocated for its earlier 3D work. This increases the risk of apps being terminated due to system memory pressure.

Test Details

Checks apps for compliance with d3d requirements and ensures that C++/XAML apps are calling a new Trim API upon their Suspend callback.

Corrective Action

The app should call the Trim API on its IDXGIDevice3 interface anytime it is about to be suspended.

App Capabilities test

Background

Special use capabilities are intended for very specific scenarios. Only company accounts are allowed to use these capabilities.

Test Details

Validate if the app is declaring any of the below capabilities:

  • EnterpriseAuthentication
  • SharedUserCertificates
  • DocumentsLibrary

If any of these capabilities are declared, the test will display a warning to the user.

Corrective Actions

Consider removing the special use capability if your app doesn't require it. Additionally, use of these capabilities are subject to additional on-boarding policy review.

Windows Runtime metadata validation

Background

Ensures that the Windows Runtime components that ship in an app conform to the Windows Runtime type system.

Test Details

Verifies that the .winmd files in the package conform to Windows Runtime rules.

Corrective Actions

  • ExclusiveTo attribute test: Ensure that Windows Runtime classes don't implement interfaces that are marked as ExclusiveTo another class.
  • Type location test: Ensure that the metadata for all Windows Runtime types is located in the winmd file that has the longest namespace-matching name in the appx package.
  • Type name case-sensitivity test: Ensure that all Windows Runtime types have unique, case-insensitive names within your appx package. Also ensure that no Windows Runtime type name is also used as a namespace name within your appx package.
  • Type name correctness test: Ensure there are no Windows Runtime types in the global namespace or in the Windows top-level namespace.
  • General metadata correctness test: Ensure that the compiler you are using to generate your Windows Runtime types is up to date with the Windows Runtime specifications.
  • Properties test: ensure that all properties on a Windows Runtime class have a get method (set methods are optional). Ensure that the type of the get method return value matches the type of the set method input parameter, for all properties on Windows Runtime types.

Please refer to Prism for the Windows Runtime reference (Windows Store business apps using C#, XAML, and Prism)

Package Sanity tests

Platform appropriate files test

Note  This test is currently only available in the latest version of the Windows App Certification Kit provided with the Windows 8.1 Preview SDK.

Apps that install mixed binaries may crash or not run correctly depending upon the user’s processor architecture.

Background

Verifies that apps do not install binaries that are not appropriate for the target processor architecture as indicated in their app manifest.

Test Details

Validates that each file's “bitness” in the PE header is appropriate when cross-referenced with the AppX manifest processor architecture declaration

Corrective Action

Ensure that binaries in the app target the appropriate processor architecture as declared in the manifest.

Supported Directory Structure test

Note  This test is currently only available in the latest version of the Windows App Certification Kit provided with the Windows 8.1 Preview SDK.

Validates that applications are not creating subdirectories as part of installation that are longer than MAX_PATH.

Background

OS components (including Trident, WWAHost, etc.) are internally limited to MAX_PATH for file system paths and will not work correctly for longer paths.

Test Details

Verifies that no path within the app install directory exceeds MAX_PATH.

Corrective Action

Use a shorter directory structure, and or file name.

Resource Usage test

Note  This test is currently only available in the latest version of the Windows App Certification Kit provided with the Windows 8.1 Preview SDK.

WinJS Background Task test

WinJS background task test ensures that JavaScript apps have the proper close statements so apps don’t consume battery.

Background

Apps that have JavaScript background tasks need to call Close() as the last statement in their background task. Apps that do not do this could keep the system from returning to connected standby mode and result in draining the battery.

Test Details

If the app does not have a background task file specified in the manifest, the test will pass. Otherwise the test will parse the JavaScript background task file that is specified in the Appx manifest, and look for a Close() statement. If found, the test will pass; otherwise the test will fail.

Corrective Action

Update the background JavaScript code to call Close() correctly.

Tests for Desktop app certification

Clean reversible install

Installs and uninstalls the app and checks for residual files and registry entries.

Background

A clean, reversible installation enables users to deploy and remove apps. To pass this test, the app must do the following.

  • The app doesn't force the system to restart immediately after installing or uninstalling the app

    An app's install or uninstall process should never require a system restart right after it completes. If this requires the system to be restarted, users should be able to restart the system at their convenience.

  • The app isn't dependent on 8.3 short file names (SFN)

    The app's install and uninstall processes must be able to use long file names and folder paths.

  • The app doesn't block silent install/uninstall

  • The app makes the required entries in the system registry

    Windows inventory tools and telemetry tools require complete info about installed apps. App installers must create the correct registry entries to allow successful detection and uninstalls.

    If you are using an MSI-based installer, MSI automatically creates the registry entries below. If you are not using an MSI installer, the installation module must create the following registry entries during installation:

    • DisplayName
    • InstallLocation
    • Publisher
    • UninstallString
    • VersionMajor or MajorVersion
    • VersionMinor or MinorVersion
  • The app must remove all of its entries in Add/Remove Programs.

Install to the correct folders test

Verifies that the app writes its program and data files to the correct folders.

Background

Apps must user system and per-user folders correctly so it can access the data and settings it needs while protecting the user's data and settings from unauthorized access.

Program Files folders

The app must be installed in the Program Files folder by default (%ProgramFiles% for native 32-bit and 64-bit apps, and %ProgramFiles(x86)% for 32-bit apps running on x64).

Note  The app must not store user data or app data in a Program Files folder because of the security permissions configured for this folder.

The ACLs on Windows system folders allow only administrator accounts to read and write to them. As a result, standard user accounts will not have access to these folders. File virtualization, however, lets apps store a file, such as a configuration file, in a system location that is typically writeable only by administrators. Running programs as a standard user in this situation could result in failure if they can't access a required file.

Apps should use Known Folders to ensure they will be able to access their data.

Note  Windows provides file virtualization to improve app compatibility and eliminate problems when apps run as a standard user on Windows. Your app should not rely on virtualization being present in future versions of Windows.

User-specific app data folders

In “per-machine” installations, the app must not write user-specific data during the installation. User-specific installation data should only be written when a user starts the app for the first time. This is because there is no correct user location at which to store data at time of installation. Attempts by an app to modify default association behaviors at a machine level after installation will be unsuccessful. Instead, defaults must be claimed on a per-user level, which prevents multiple users from overwriting each other's defaults.

All app data exclusive to a specific user and not to be shared with other users of the computer must be stored in Users\<username>\AppData.

All app data that must be shared among users on the computer should be stored within ProgramData.

Other system folders and registry keys

The app should never write directly to the Windows directory and or subdirectories. Use the correct methods for installing files, such as fonts or drivers, to these directories.

Apps should not start automatically on startup, such as by adding an entry to one or more of these locations:

  • Registry run keys HKLM and, or HKCU under Software\Microsoft\Windows\CurrentVersion
  • Registry run keys HKLM, and or HKCU under Software\Wow6432Node\Microsoft\windows\CurrentVersion
  • Start Menu AllPrograms > STARTUP

Test details

This test verifies that the app uses the specific locations in the file system that Windows provides to store programs and software components, shared app data, and app data that is specific to a user.

Exceptions and Waivers

Note  Applies to desktop app certification only.

A waiver is required for apps writing to the global assembly cache (GAC) (.NET apps should keep assembly dependencies private, and store it in the app's directory unless sharing an assembly is explicitly required).

Installation to the Programs Files folder is not a requirement for Desktop SW packages to achieve SW Logo, only under the SW Fundamentals category.

Corrective actions

Review how the app uses the system's folders and make sure it's using them correctly.

Test details

This test checks the app's install and uninstall processes for the required behavior described in Background.

Corrective action

Review the app's design and behavior against the requirements described in Background.

Digitally signed file test

Tests executable files and device drivers to verify they have a valid digital signature.

Background

Digitally signed files make it possible to verify the file is genuine and to detect if the file has been tampered with.

Kernel-mode code signing enforcement is a Windows feature that is also known as code integrity (CI). CI improves the security of Windows by verifying the integrity of a file each time it is loaded into memory. CI detects whether malicious code has modified a system binary file and generates a diagnostic and system-audit log event when the signature of a kernel module fails to verify correctly.

If the app requires elevated permissions, the elevation prompt displays contextual info about the executable file that is requesting elevated access. Depending on whether the app is Authenticode signed, the user may see the consent prompt or the credential prompt.

Strong names prevent third parties from spoofing your code, as long as you keep the private key secure. The .NET Framework verifies the digital signature either when it loads the assembly or installs it in the GAC. Without access to the private key, a malicious user cannot modify your code and sign it again.

Test details

All executable files, such as those with file extensions of .exe, .dll, .ocx, .sys, .cpl, .drv, and .scr, must be signed with an Authenticode certificate.

All kernel mode drivers installed by the app must have a Microsoft signature obtained through the WHQL or DRS program. All file system filter drivers must be WHQL signed.

Corrective action

Sign the app’s executable files.

Use makecert.exe to generate a certificate or obtain a code-signing key from one of the commercial certification authorities (CAs), such as VeriSign, Thawte, or a Microsoft CA.

Exceptions and waivers

Note  Applies to desktop app certification only.

Waivers will be considered only for unsigned third-party redistributables. A proof of communication requesting a signed version of the redistributable(s) is required for this waiver to be granted.

Device value-added software packages are exempt from the kernel mode driver certification, because drivers must be certified in by Windows Hardware Certification.

Support x64 Windows test

Test the app to make sure the .exe is built for the platform architecture onto which it will be installed.

Background

The executable file must be built for the processor architecture on which it is installed. Some executable files might run on a different processor architecture, but this is not reliable.

Architecture compatibility is important because 32-bit processes cannot load 64-bit DLLs and 64-bit processes cannot load 32-bit DLLs. Likewise, 64-bit versions of Windows does not support running 16-bit Windows-based applications because handles have 32 significant bits on 64-bit Windows an so they can't be passed to 16-bit applications. Therefore, trying to launch a 16-bit application will fail on 64-bit versions of Windows.

32-bit device drivers cannot run on 64-bit versions of Windows and therefore, they must be ported to the 64-bit architecture.

For user-mode applications, 64-bit Windows includes WOW64, which enables 32-bit Windows applications to execute on systems running 64-bit Windows, albeit with some loss of performance. No equivalent translation layer exists for device drivers.

To maintain compatibility with 64-bit versions of Windows, apps must natively support 64-bit or, at a minimum, 32-bit Windows-based applications must run seamlessly on 64-bit systems:

  • Apps and their installers mustn't contain any 16-bit code or rely on any 16-bit component.
  • App setup must detect and install the proper drivers and components on 64-bit versions of Windows.
  • Any shell plug-ins must run on 64-bit versions of Windows.
  • Apps that run under the WoW64 emulator should not attempt to bypass Wow64 virtualization mechanisms. If there are specific scenarios where apps need to detect if they are running in a WoW64 emulator, they should do so by calling IsWow64Process.

OS version checking test

Tests how the app checks for the version of Windows on which it's running.

Background

Apps check the OS version by testing for a version that is greater than or equal to the required version to ensure compatibility with future versions of Windows.

Test details

Simulates running the app on different versions of Windows to see how it reacts.

Exceptions and Waivers

Note  Applies to desktop app certification only.

Waivers will be considered for apps that meet the following criteria:

  • Apps that are delivered as one package that runs on Windows XP, Windows Vista, and Windows 7 and need to check the OS version to determine which components to install on a given operating system.

  • Apps that check only the minimum version of the OS (during install only, not at runtime) by using only the approved API calls and list the minimum version requirement in the app manifest as required.

  • Security apps such as antivirus and firewall apps, system utilities such as defrag utilities and backup apps, and diagnostics tools that check the OS version by using only the approved API calls.

Corrective actions

Test for the correct version of Windows by testing if the current version is greater-than or equal to the version your app, service, or driver needs.

Driver installers and uninstall modules must never check the OS version.

User account control (UAC) test

Tests the app to verify that it doesn't need unnecessarily elevated permissions to run.

Background

An app that operates or installs only when the user is an administrator forces users to run the app with unnecessarily elevated permissions, which can allow malware to enter the user's computer.

When users are always forced to run applications with elevated access tokens, the app can server as an entry point for deceptive or malicious code. This malware can easily modify the operating system, or worse, affect other users. It is nearly impossible to control a user that has full administrator access, because Administrators can install applications and run any application or script on the computer. IT managers are always seeking ways to create "standard desktops" where users log on as standard users. Standard desktops greatly reduce help desk costs and reduce IT overhead.

Most applications don't require administrator privileges at run time. A standard-user account should be able to run them. Windows apps must have a manifest (embedded or external) that defines execution levels and tells Windows what privileges the application requires to run. The app manifest only applies to .exe files, not .dll files. User Account Control (UAC) does not inspect DLLs during the creation of the process. UAC rules don't apply to Microsoft services. The app manifest can be embedded or external.

To create a manifest, create a file with the name <app_name>.exe.manifest and store it in the same directory as the EXE. Note that any external manifest is ignored if the application has an internal manifest.

  • For example, <requestedExecutionLevel level=""asInvoker | highestAvailable | requireAdministrator"" uiAccess=""true|false""/>

  • The main process of the application must be run as a standard user (asInvoker). Any administrative features must be moved into a separate process that runs with administrative privileges.

  • User facing applications that require elevated privileges must be Authenticode signed.

Exceptions and waivers

Note  Applies to desktop app certification only.

A waiver is required for apps that run their main process with elevated privileges (requireAdministrator or highestAvailable). The main process is the process that provides the user’s entry point to the application.

Waivers will be considered for the following scenarios:

●  Administrative or system tools with execution level set to highestAvailable, requireAdministrator , or both.

Or

● Only Accessibility or UI automation framework application sets the uiAccess flag to TRUE to bypass the user interface privilege isolation (UIPI). To properly start app utilization, this flag must be Authenticode signed, and must reside in a protected location in the file system, such as Program Files.

Corrective actions

Review the app's manifest file for the correct entries and permission levels.

Adhere to system restart manager messages

Tests how the app responds to system shutdown and restart messages.

Background

Apps must exit as quickly as possible when they are notified the system is shutting down to provide a responsive shutdown or power-off experience for the user.

In a critical shutdown, apps that return FALSE to WM_QUERYENDSESSION will be sent WM_ENDSESSION and closed, while those that time out in response to WM_QUERYENDSESSION will be forcibly terminated.

Test details

Examines how the app responds to shutdown and exit messages.

Corrective actions

If your app fails this test, review how it handles these Windows messages.

WM_QUERYENDSESSION with LPARAM = ENDSESSION_CLOSEAPP(0x1): Desktop apps must respond (TRUE) immediately in preparation for a restart. Console apps can call SetConsoleCtrlHandler to receive shutdown notification. Services can call RegisterServiceCtrlHandlerEx to receive shutdown notifications in a handler routine.

WM_ENDSESSION with LPARAM = ENDSESSION_CLOSEAPP(0x1): Apps must return a 0 value within 30 seconds and shut down. At a minimum, apps should prepare by saving any user data and state the info that is needed after a restart.

Console apps that receive the CTRL_C_EVENT notification should shut down immediately. Drivers must not veto a system shutdown event.

Note  Apps that must block shutdown because of an operation that cannot be interrupted should use ShutdownBlockReasonCreate to register a string that explains the reason to the user. When the operation has completed, the app should call ShutdownBlockReasonDestroy to indicate that the system can be shut down.

Safe mode test

Tests if the driver or service is configured to start in safe mode.

Background

Safe mode allows users to diagnose and troubleshoot problems with Windows. Only drivers and services that are necessary to the basic operation of the operating system or provide diagnostic and recovery services should load in safe mode. Loading other files in safe mode makes it more difficult to troubleshoot problems with the operating system.

By default, only the drivers and services that come pre-installed with Windows start in safe mode. All other drivers and services should be disabled unless the system requires them for basic operations or for diagnostic and recovery purposes.

Exceptions and waivers

Note  Applies to desktop app certification only.

Drivers and services that must start in safe mode require a waiver to be certified. The waiver request must include each driver and service to add to the SafeBoot registry keys and describe the technical reasons for why the driver or service must run in safe mode. The app installer must register all such drivers and services in these registry keys:

  • HKLM/System/CurrentControlSet/Control/SafeBoot/Minimal
  • HKLM/System/CurrentControlSet/Control/SafeBoot/Network

Note  

You must test the drivers and services you want to start in safe mode to ensure that they function in safe mode without any errors.

Corrective actions

If the driver or service should not start in safe mode, remove the app's entries from the registry keys.

Corrective actions

Build the executable files and drivers for the processor architecture for which you want to install them.

Multiuser session test

Test how the app behaves when run in multiple sessions at the same time.

Background

Windows users must be able to run concurrent sessions. Apps must ensure that when they run in multiple sessions, either locally or remotely, the normal functionality of the app is not adversely affected. App settings and data files must be user-specific and the user's privacy and preferences must be restricted to the user's session.

Test details

Runs multiple concurrent instances of the app to test the following:

  • Multiple instances of an app running at the same time are isolated from each other.

    This means that user data from one instance is not visible to another instance. Sound in an inactive user session should not be heard in an active user session. In cases where multiple app instances use shared resources, the app must ensure that there is not a conflict.

  • If the app was installed for multiple users, it stores data in the correct folder(s) and registry locations.

  • The app can run in multiple user sessions (Fast User Switching) for both local and remote access.

    To ensure this, the app must check other terminal service (TS) sessions for existing instances of the app. If the app does not support multiple user sessions or remote access, it must clearly say this to the user when it's launched from such a session.

Corrective action

Make sure that the app doesn’t store system-wide data files or settings in user-specific data stores, such as User Profile or HKCU. If it does, that info won’t be available to other users.

Your app must install system-wide configuration and data files during installation and create the user-specific files and settings after installation when a user runs it.

Make sure that the app doesn’t block multiple concurrent sessions, either locally or remotely. The app must not depend on global mutexes or other named-objects to check for or block multiple concurrent sessions.

If the app can’t allow multiple concurrent sessions per user, use per-user or per-session namespaces for mutexes and other named-objects.

Crashes and hangs test

Monitors the app during certification testing to record when it crashes or hangs.

Background

App failures such as crashes or hangs are a major disruption to the user experience and risk a potential loss of user data. Eliminating these failures improves app stability, reliability, and provides users with a better experience.

Apps that stop responding or crash can cause the user to lose data and have a poor experience.

Test details

We test the app resilience and stability throughout the certification testing.

The Windows App Certification Kit calls IApplicationActivationManager::ActivateApplication to launch Windows Store apps. For ActivateApplication to launch an app, User Account Control (UAC) must be enabled and the screen resolution must be at least 1024 x 768 or 768 x 1024. If either condition is not met, your app will fail this test.

Corrective actions

Make sure UAC is enabled on the test computer.

Make sure you are running the test on a computer with large enough screen.

If your app fails to launch and your test platform satisfies the prerequisites of ActivateApplication, you can troubleshoot the problem by reviewing the activation event log. To find these entries in the event log:

  1. Open eventvwr.exe and navigate to the Application and Services Log\Microsoft\Windows\Immersive-Shell folder.
  2. Filter the view to show Event Ids: 5900-6000.
  3. Review the log entries for info that might explain why the app didn't launch.

Troubleshoot the file with the problem, identify and fix the problem. Rebuild and re-test the app.

Additional resources

Windows Vista and Windows 7 Operating Systems

Using Application Verifier Within Your Software Development Lifecycle

Application Verifier

Using Application Verifier

AppInit DLLs

Minimize startup time (Windows Store apps using C#/VB/C++ and XAML)

Compatibility and resiliency test

Background

Apps must be fully functional without using Windows compatibility modes, AppHelp messages, or other compatibility fixes. This check will validate two aspects of an app: the app main executable (user facing app entry point) must be manifested for compatibility, as well as declaring the right GUID. To support this new test the report will have a sub node under ‘Compatibility & resiliency’. The app will fail if one or both of these conditions are missing.

Test Details

  • Compatibility: Apps must be fully functional without using Windows compatibility modes, AppHelp messages, or other compatibility fixes. A compatibility manifest allows Windows to provide your app the proper compatibility behavior under the different versions of the OS.
  • AppInit: Apps must not list DLLs to load in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs registry key.
  • Switchback: The application must provide the switchback manifest. If the manifest is missing, Windows App Certification Kit gives a warning message. It will also verify the manifest contains valid OS GUID.

Corrective actions

  • Fix the app's component that uses the compatibility fix.

  • Make sure that the app doesn’t rely on compatibility fixes for its functionality.

  • Ensure your app is manifested and that the compatibility section includes the appropriate values.

See AppInit DLLs for more info.

Windows Security best practices test

Background

An application should not change the default Windows security settings.

Test Details

Categories of failure on a per test basis. For example some categories for security test can include:

  • Initialization failure
  • Security Architecture failure
  • Possible Buffer overflow failure
  • Crash failure

Windows Security features test

Background

Applications must opt-into Windows security features. Changing the default Windows security protections can put customers at increased risk.

Test Details

Tests the app's security by running the BinScope Binary Analyzer and the Attack Surface Analyzer.

Corrective Actions

Troubleshoot and fix the problem identified by the tests. Rebuild and re-test the app.

High DPI test

It is highly recommended for Win32 applications to be DPI aware. It is the key to make the application UI look consistently good across a wide variety of high-DPI display settings. A non-DPI-aware application running on a high-DPI display setting may have problems such as incorrect scaling of UI elements, clipped text, and blurry images. There are two ways to declare an application is DPI aware. One is to declare DPI.

Background

This check will validate two aspects of an app: the main executable (user facing app entry point) must be manifested for high-DPI awareness and that the proper APIs are being called to support high-DPI. The app will fail if one or both of these conditions are missing. This check will introduce a new section in the desktop report.

Test Details

Test will generate a warning when any of the following are detected:

  • The main EXE doesn’t declare DPI awareness in its manifest and it doesn’t call SetProcessDPIAware API (Warn the developer if he forgets to add DPI awareness manifest).
  • The main EXE doesn’t declare DPI awareness in its manifest, but it calls SetProcessDPIAware API (Warn the developer that he should use manifest to declare DPI awareness instead of calling the API).
  • The main EXE declares DPI awareness in its manifest, but it also calls SetProcessDPIAware API (Warn the developer that calling that API is not necessary).
  • For binaries that are not main EXEs, a warning is given if they calls the API (Calling the API is not recommended in this situation).

Corrective Actions

The use of SetProcessDPIAware() function is discouraged. If a DLL caches DPI settings during its initialization, invoking SetProcessDPIAware() in the application may generate a race condition. Calling SetProcessDPIAware() function in a DLL is also not a recommended practice.

See Writing High-DPI apps for more info.

 

 

Build date: 7/1/2013

© 2013 Microsoft. All rights reserved.