Windows App Certification Kit tests

[This topic is featured in Develop great apps for Windows 8.]

The Windows App Certification Kit contains a number of tests that can help ensure that your app is ready to be published on the Windows Store.

Adhere to system restart manager messages

Tests how the app responds to system shutdown and restart messages.

Background

Apps must exit as quickly as possible when they are notified the system is shutting down to provide a responsive shutdown or power-off experience for the user.

In a critical shutdown, apps that return FALSE to WM_QUERYENDSESSION will be sent WM_ENDSESSION and closed, while those that time out in response to WM_QUERYENDSESSION will be forcibly terminated.

Test details

Examines how the app responds to shutdown and exit messages.

Corrective actions

If your app fails this test, review how it handles these Windows messages.

WM_QUERYENDSESSION with LPARAM = ENDSESSION_CLOSEAPP(0x1): Desktop apps must respond (TRUE) immediately in preparation for a restart. Console apps can call SetConsoleCtrlHandler to receive shutdown notification. Services can call RegisterServiceCtrlHandlerEx to receive shutdown notifications in a handler routine.

WM_ENDSESSION with LPARAM = ENDSESSION_CLOSEAPP(0x1): Apps must return a 0 value within 30 seconds and shut down. At a minimum, apps should prepare by saving any user data and state the info that is needed after a restart.

Console apps that receive the CTRL_C_EVENT notification should shut down immediately. Drivers must not veto a system shutdown event.

Note  Apps that must block shutdown because of an operation that cannot be interrupted should use ShutdownBlockReasonCreate to register a string that explains the reason to the user. When the operation has completed, the app should call ShutdownBlockReasonDestroy to indicate that the system can be shut down.

App manifest policy test

Test the contents of app manifest to make sure its contents are correct.

Background

Apps must have a correctly formatted app manifest.

Test details

Examines the app manifest to verify the contents are correct as described in the Package metadata requirements.

Corrective action

Review the app's manifest against the requirements described in the Package metadata requirements.

App manifest resources test

Tests the resources defined in the app manifest to make sure they are present and valid.

Background

The app might not install if the strings or images declared in the app’s manifest are incorrect. If the app installs with these errors, the app’s logo or other images used by the app might not display correctly.

Test details

Inspects the resources defined in the app manifest to make sure they are present and valid.

Corrective actions

Fix the problem identified in the error message.

Clean reversible install

Installs and uninstalls the app and checks for residual files and registry entries.

Background

A clean, reversible installation enables users to deploy and remove apps. To pass this test, the app must do the following.

• The app doesn't force the system to restart immediately after installing or uninstalling the app

  An app's install or uninstall process should never require a system restart right after it completes. If this requires the system to be restarted, users should be able to restart the system at their convenience.

• The app isn't dependent on 8.3 short file names (SFN)

  The app's install and uninstall processes must be able to use long file names and folder paths.

• The app doesn't block silent install/uninstall

• The app makes the required entries in the system registry

  Windows inventory tools and telemetry tools require complete info about installed apps. App installers must create the correct registry entries to allow successful detection and uninstalls.

  If you are using an MSI-based installer, MSI automatically creates the registry entries below. If you are not using an MSI installer, the installation module must create the following registry entries during installation:

  • DisplayName
  • InstallLocation
  • Publisher
  • UninstallString
  • VersionMajor or MajorVersion
  • VersionMinor or MinorVersion
• The app must remove all of its entries in Add/Remove Programs.

Test details

This test checks the app's install and uninstall processes for the required behavior described in Background.

Corrective action

Review the app's design and behavior against the requirements described in Background.

Compatibility and resiliency test

Verifies the app doesn't use any Windows compatibility fixes.

Background

Apps must be fully functional without using Windows compatibility modes, AppHelp messages, or other compatibility fixes.

Corrective actions

Fix the app's component that uses the compatibility fix.

Make sure that the app doesn’t rely on compatibility fixes for its functionality.

Crashes and hangs test

Monitors the app during certification testing to record when it crashes or hangs.

Background

Apps that stop responding or crash can cause the user to lose data and have a poor experience.

We expect apps to be fully functional without the use of Windows compatibility modes, AppHelp messages, or compatibility fixes.

Apps must not list DLLs to load in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs registry key.

Test details

We test the app resilience and stability throughout the certification testing.

The Windows App Certification Kit calls IApplicationActivationManager::ActivateApplication to launch Windows Store apps. For ActivateApplication to launch an app, User Account Control (UAC) must be enabled and the screen resolution must be at least 1024 x 768 or 768 x 1024. If either condition is not met, your app will fail this test.

Corrective actions

Make sure UAC is enabled on the test computer.

Make sure you are running the test on a computer with large enough screen.

If your app fails to launch and your test platform satisfies the prerequisites of ActivateApplication, you can troubleshoot the problem by reviewing the activation event log. To find these entries in the event log:

1. Open eventvwr.exe and navigate to the Application and Services Log\Microsoft\Windows\Immersive-Shell folder.
2. Filter the view to show Event Ids: 5900-6000.
3. Review the log entries for info that might explain why the app didn't launch.

Troubleshoot the file with the problem, identify and fix the problem. Rebuild and re-test the app.

Additional resources

Windows Vista and Windows 7 Operating Systems

Using Application Verifier Within Your Software Development Lifecycle

Application Verifier

Using Application Verifier

AppInit DLLs

Minimize startup time (Windows Store apps using C#/VB/C++ and XAML)

Debug configuration test

Test the app to make sure it is not a debug build.

Background

To be certified for the Windows Store, apps must not be compiled for debug and they must not reference debug versions of an executable file. In addition, you must build your code as optimized for your app to pass this test.

Test details

Test the app to make sure it is not a debug build and is not linked to any debug frameworks.

Corrective actions

Build the app as a release build before you submit it to the Windows Store. Make sure that you have the correct version of .NET framework installed.

Digitally signed file test

Tests executable files and device drivers to verify they have a valid digital signature.

Background

Digitally signed files make it possible to verify the file is genuine and to detect if the file has been tampered with.

Kernel-mode code signing enforcement is a Windows feature that is also known as code integrity (CI). CI improves the security of Windows by verifying the integrity of a file each time it is loaded into memory. CI detects whether malicious code has modified a system binary file and generates a diagnostic and system-audit log event when the signature of a kernel module fails to verify correctly.

If the app requires elevated permissions, the elevation prompt displays contextual info about the executable file that is requesting elevated access. Depending on whether the app is Authenticode signed, the user may see the consent prompt or the credential prompt.

Strong names prevent third parties from spoofing your code, as long as you keep the private key secure. The .NET Framework verifies the digital signature either when it loads the assembly or installs it in the GAC. Without access to the private key, a malicious user cannot modify your code and sign it again.

Test details

All executable files, such as those with file extensions of .exe, .dll, .ocx, .sys, .cpl, .drv, and .scr, must be signed with an Authenticode certificate.

All kernel mode drivers installed by the app must have a Microsoft signature obtained through the WHQL or DRS program. All file system filter drivers must be WHQL signed.

Corrective action

Sign the app’s executable files.

Use makecert.exe to generate a certificate or obtain a code-signing key from one of the commercial certification authorities (CAs), such as VeriSign, Thawte, or a Microsoft CA.

Exceptions and waivers

Note  Applies to desktop app certification only.

Waivers will be considered only for unsigned third-party redistributables. A proof of communication requesting a signed version of the redistributable(s) is required for this waiver to be granted.

Device value-added software packages are exempt from the kernel mode driver certification, because drivers must be certified in by Windows Hardware Certification.

File encoding test

Test the contents of app packages to make sure they use the correct file encoding.

Background

App packages must use the correct file encoding.

Test details

{Test Details}

Exceptions and waivers

Note  Applies to desktop app certification only.

{Exceptions and Waivers}

Install to the correct folders test

Verifies that the app writes its program and data files to the correct folders.

Background

Apps must user system and per-user folders correctly so it can access the data and settings it needs while protecting the user's data and settings from unauthorized access.

Program Files folders

The app must be installed in the Program Files folder by default (%ProgramFiles% for native 32-bit and 64-bit apps, and %ProgramFiles(x86)% for 32-bit apps running on x64).

Note  The app must not store user data or app data in a Program Files folder because of the security permissions configured for this folder.

The ACLs on Windows system folders allow only administrator accounts to read and write to them. As a result, standard user accounts will not have access to these folders. File virtualization, however, lets apps store a file, such as a configuration file, in a system location that is typically writeable only by administrators. Running programs as a standard user in this situation could result in failure if they can't access a required file.

Apps should use Known Folders to ensure they will be able to access their data.

Note  Windows provides file virtualization to improve app compatibility and eliminate problems when apps run as a standard user on Windows. Your app should not rely on virtualization being present in future versions of Windows.

User-specific app data folders

In “per-machine” installations, the app must not write user-specific data during the installation. User-specific installation data should only be written when a user starts the app for the first time. This is because there is no correct user location at which to store data at time of installation. Attempts by an app to modify default association behaviors at a machine level after installation will be unsuccessful. Instead, defaults must be claimed on a per-user level, which prevents multiple users from overwriting each other's defaults.

All app data exclusive to a specific user and not to be shared with other users of the computer must be stored in Users\<username>\AppData.

All app data that must be shared among users on the computer should be stored within ProgramData.

Other system folders and registry keys

The app should never write directly to the Windows directory and or subdirectories. Use the correct methods for installing files, such as fonts or drivers, to these directories.

Apps should not start automatically on startup, such as by adding an entry to one or more of these locations:

• Registry run keys HKLM and, or HKCU under Software\Microsoft\Windows\CurrentVersion
• Registry run keys HKLM, and or HKCU under Software\Wow6432Node\Microsoft\windows\CurrentVersion
• Start Menu AllPrograms > STARTUP

Test details

This test verifies that the app uses the specific locations in the file system that Windows provides to store programs and software components, shared app data, and app data that is specific to a user.

Exceptions and Waivers

Note  Applies to desktop app certification only.

A waiver is required for apps writing to the global assembly cache (GAC) (.NET apps should keep assembly dependencies private, and store it in the app's directory unless sharing an assembly is explicitly required).

Installation to the Programs Files folder is not a requirement for Desktop SW packages to achieve SW Logo, only under the SW Fundamentals category.

Corrective actions

Review how the app uses the system's folders and make sure it's using them correctly.

Multiuser session test

Test how the app behaves when run in multiple sessions at the same time.

Background

Windows users must be able to run concurrent sessions. Apps must ensure that when they run in multiple sessions, either locally or remotely, the normal functionality of the app is not adversely affected. App settings and data files must be user-specific and the user's privacy and preferences must be restricted to the user's session.

Test details

Runs multiple concurrent instances of the app to test the following:

• Multiple instances of an app running at the same time are isolated from each other.

  This means that user data from one instance is not visible to another instance. Sound in an inactive user session should not be heard in an active user session. In cases where multiple app instances use shared resources, the app must ensure that there is not a conflict.

• If the app was installed for multiple users, it stores data in the correct folder(s) and registry locations.

• The app can run in multiple user sessions (Fast User Switching) for both local and remote access.

  To ensure this, the app must check other terminal service (TS) sessions for existing instances of the app. If the app does not support multiple user sessions or remote access, it must clearly say this to the user when it's launched from such a session.

Corrective action

Make sure that the app doesn’t store system-wide data files or settings in user-specific data stores, such as User Profile or HKCU. If it does, that info won’t be available to other users.

Your app must install system-wide configuration and data files during installation and create the user-specific files and settings after installation when a user runs it.

Make sure that the app doesn’t block multiple concurrent sessions, either locally or remotely. The app must not depend on global mutexes or other named-objects to check for or block multiple concurrent sessions.

If the app can’t allow multiple concurrent sessions per user, use per-user or per-session namespaces for mutexes and other named-objects.

OS version checking test

Tests how the app checks for the version of Windows on which it's running.

Background

Apps check the OS version by testing for a version that is greater than or equal to the required version to ensure compatibility with future versions of Windows.

Test details

Simulates running the app on different versions of Windows to see how it reacts.

Exceptions and Waivers

Note  Applies to desktop app certification only.

Waivers will be considered for apps that meet the following criteria:

• Apps that are delivered as one package that runs on Windows XP, Windows Vista, and Windows 7 and need to check the OS version to determine which components to install on a given operating system.

• Apps that check only the minimum version of the OS (during install only, not at runtime) by using only the approved API calls and list the minimum version requirement in the app manifest as required.

• Security apps such as antivirus and firewall apps, system utilities such as defrag utilities and backup apps, and diagnostics tools that check the OS version by using only the approved API calls.

Corrective actions

Test for the correct version of Windows by testing if the current version is greater-than or equal to the version your app, service, or driver needs.

Driver installers and uninstall modules must never check the OS version.

Performance test

Tests how quickly an app launches when the user starts it, how fast it suspends when the user switches to another app and the peak memory usage, or working set, that the app uses.

Background

The app must respond quickly to user interaction and system commands from Windows to present a fast and fluid experience to the user.

Test details

The characteristics of the computer on which the test is performed can influence the test results. The performance test thresholds for app certification are set such that low-power computers meet the customer’s expectation of fast and fluid experience. To determine your app’s performance, we recommend that you test on a low-power computer, such as an Intel Atom processor-based computer with a screen resolution of 1366x768 (or higher) and a rotational hard drive (as opposed to a solid-state hard drive).

• Launch and suspend. The time an app takes depends on many variables, such as system load and configuration, which can vary from one computer to another. Because of this variability, a launch time or a suspend time that is just under the limit in one instance, could be just over the limit, and fail the test, in another. If your app’s launch time or suspend time is close to the limit, review what the app is doing at these times and look for ways to reduce that activity.
• Peak memory usage. Efficient use of memory is critical for apps to deliver a fast and fluid experience. Apps that consume a large amount of memory will also negatively impact other user experiences on the system. In extreme cases the apps will not run on machines with 1GB of RAM. To accurately measure peak memory usage (also known as peak working set), key scenarios must be tested while collecting memory usage data. This prevents this test from being fully automated in the Windows App Certification Kit. Even with this limitation, this test is still part of the Windows Store acceptance criteria. Thus, we strongly recommend that you test this using the process described in the Frequently Asked Questions section before submission.

As low-power computers evolve, their performance characteristics might change over time. Test your app with the most current version of the Windows App Certification Kit to make sure that your app complies with the latest performance requirements.

Corrective action

For general advice on launching and suspending apps written in JavaScript, see Optimizing your app's lifecycle and Performance best practices for Windows Store apps using JavaScript for an overview of best practices. If you are having difficulty meeting the load time limit and are writing in JavaScript, see Reducing your app's loading time for additional advice on how to build your app. C# developers, check out similar info in Minimize startup time.

When you run the Windows App Certification Kit, make sure that nothing else is running on the machine, as this will affect the results of the performance test. We don't recommend running the test on a remote desktop or in a VM, as these will also have a negative impact on the performance of your test.

This table has guidance for resolving Windows App Certification Kit errors from the launch and suspend test.

Frequently asked questions

What are the performance limits for the launch test?

Here are the limits for the launch test:

Why do I get a warning when my app is launching in under 5 seconds?

Computers using an x86 or x64 processor are more powerful that low-power computers. An app that launches in 3 seconds on these computers may exceed the 5 second limit on a low-power system. The warning informs you that your app is at risk of failing windows store certification. If possible, test on a low-power computer.

What are the performance limits for the suspend test?

All computers have the same limits:

• PASS: less than or equal to 2 seconds
• FAIL: greater than 2 seconds

How do I measure the peak memory working set

The Task Manager provides a very convenient way to see the working set of your app during execution. Before you run the app, press Crtl+Alt+Delete and open the Task Manager. Select More details and open the processes tab. In the Options tab select Always on top so the Task Manager remains visible. The working set is shown in the Memory column.

To pass the Windows Store Performance test, your app must not exceed a working set of 500MB. To determine if your app meets this requirement, open it and run it through different scenarios to see how the app uses memory. As you run through the scenarios, keep an eye on the Memory column in the Task Manager. As the memory needs of your app change, the value in that column goes up or down. Keep in mind that if the Task Manager window is selected your app will lose focus which may affect the test. You can also use the Windows Performance Analysis tool to learn more about your app's memory usage and other performance issues.

How is performance measured?

Launch and Suspend durations are calculated from the ETW events generated for the app under test. The time is calculated from the time difference between the Start and Stop event pairs of ETW events from the Microsoft-Windows-Immersive-Shell provider identified below.

Launch Test:

• "PerfTrack_SplashScreen_AppShown/win:Start"
• "PerfTrack_SplashScreen_AppShown/win:Stop"

Suspend Test:

• "PerfTrack_PLM_SuspendPackage/win:Start"
• "PerfTrack_PLM_SuspendPackage/win:Stop"

Safe mode test

Tests if the driver or service is configured to start in safe mode.

Background

Safe mode allows users to diagnose and troubleshoot problems with Windows. Only drivers and services that are necessary to the basic operation of the operating system or provide diagnostic and recovery services should load in safe mode. Loading other files in safe mode makes it more difficult to troubleshoot problems with the operating system.

By default, only the drivers and services that come pre-installed with Windows start in safe mode. All other drivers and services should be disabled unless the system requires them for basic operations or for diagnostic and recovery purposes.

Exceptions and waivers

Note  Applies to desktop app certification only.

Drivers and services that must start in safe mode require a waiver to be certified. The waiver request must include each driver and service to add to the SafeBoot registry keys and describe the technical reasons for why the driver or service must run in safe mode. The app installer must register all such drivers and services in these registry keys:

• HKLM/System/CurrentControlSet/Control/SafeBoot/Minimal
• HKLM/System/CurrentControlSet/Control/SafeBoot/Network

Note  

You must test the drivers and services you want to start in safe mode to ensure that they function in safe mode without any errors.

Corrective actions

If the driver or service should not start in safe mode, remove the app's entries from the registry keys.

Support x64 Windows test

Test the app to make sure the .exe is built for the platform architecture onto which it will be installed.

Background

The executable file must be built for the processor architecture on which it is installed. Some executable files might run on a different processor architecture, but this is not reliable.

Architecture compatibility is important because 32-bit processes cannot load 64-bit DLLs and 64-bit processes cannot load 32-bit DLLs. Likewise, 64-bit versions of Windows does not support running 16-bit Windows-based applications because handles have 32 significant bits on 64-bit Windows an so they can't be passed to 16-bit applications. Therefore, trying to launch a 16-bit application will fail on 64-bit versions of Windows.

32-bit device drivers cannot run on 64-bit versions of Windows and therefore, they must be ported to the 64-bit architecture.

For user-mode applications, 64-bit Windows includes WOW64, which enables 32-bit Windows applications to execute on systems running 64-bit Windows, albeit with some loss of performance. No equivalent translation layer exists for device drivers.

To maintain compatibility with 64-bit versions of Windows, apps must natively support 64-bit or, at a minimum, 32-bit Windows-based applications must run seamlessly on 64-bit systems:

• Apps and their installers mustn't contain any 16-bit code or rely on any 16-bit component.
• App setup must detect and install the proper drivers and components on 64-bit versions of Windows.
• Any shell plug-ins must run on 64-bit versions of Windows.
• Apps that run under the WoW64 emulator should not attempt to bypass Wow64 virtualization mechanisms. If there are specific scenarios where apps need to detect if they are running in a WoW64 emulator, they should do so by calling IsWow64Process.

Corrective actions

Build the executable files and drivers for the processor architecture for which you want to install them.

Supported Windows Store API test

Test the app for the use of any non-compliant APIs.

Background

Apps must use the APIs for Windows Store apps to be certified for the Windows Store.

Test details

Tests the executable elements of the app to verify they call only APIs for Windows Store apps.

Corrective actions

Make sure that the app was compiled as a release build and not a debug build.

Note  The debug build of an app will fail this test even if the app uses only APIs for Windows Store apps.

Review the error messages to identify the API the app uses that is not an API for Windows Store apps.

User account control (UAC) test

Tests the app to verify that it doesn't need unnecessarily elevated permissions to run.

Background

An app that operates or installs only when the user is an administrator forces users to run the app with unnecessarily elevated permissions, which can allow malware to enter the user's computer.

When users are always forced to run applications with elevated access tokens, the app can server as an entry point for deceptive or malicious code. This malware can easily modify the operating system, or worse, affect other users. It is nearly impossible to control a user that has full administrator access, because Administrators can install applications and run any application or script on the computer. IT managers are always seeking ways to create "standard desktops" where users log on as standard users. Standard desktops greatly reduce help desk costs and reduce IT overhead.

Most applications don't require administrator privileges at run time. A standard-user account should be able to run them. Windows apps must have a manifest that defines execution levels and tells Windows what privileges the application requires to run. The app manifest only applies to .exe files, not .dll files. User Account Control (UAC) does not inspect DLLs during the creation of the process. UAC rules don't apply to Windows services. The app manifest can be embedded or external.

To create a manifest, create a file with the name <app_name>.exe.manifest and store it in the same directory as the EXE. Note that any external manifest is ignored if the application has an internal manifest.

• For example, <requestedExecutionLevel level=""asInvoker | highestAvailable | requireAdministrator"" uiAccess=""true|false""/>

• The main process of the application must be run as a standard user (asInvoker). Any administrative features must be moved into a separate process that runs with administrative privileges.

• User facing applications that require elevated privileges must be Authenticode signed.

Test details

{Test details}

Exceptions and waivers

Note  Applies to desktop app certification only.

A waiver is required for apps that run their main process with elevated privileges (requireAdministrator or highestAvailable). The main process is the process that provides the user’s entry point to the application.

Waivers will be considered for the following scenarios:

●  Administrative or system tools with execution level set to highestAvailable, requireAdministrator , or both.

Or

● Only Accessibility or UI automation framework application sets the uiAccess flag to TRUE to bypass the user interface privilege isolation (UIPI). To properly start app utilization, this flag must be Authenticode signed, and must reside in a protected location in the file system, such as Program Files.

Corrective actions

Review the app's manifest file for the correct entries and permission levels.

Review The Windows Vista and Windows Server 2008 Developer Story: Windows Vista Application Development Requirements for User Account Control (UAC).

Direct3d feature level test

Tests Microsoft Direct3D apps to ensure that they work on all Windows 8 graphics hardware.

Background

Direct3D apps must support the minimum Direct3D feature level that they specify. See the Direct3D certification requirement for details.

Test details

Runs the app at Direct3D feature level 9_1 to check that the app launches and runs correctly.

Corrective action

Ensure that the app correctly supports the minimum Direct3D feature level you chose. If that feature level is higher than feature level 9_1, then your app must still run correctly at feature level 9_1. See Developing for different Direct3D feature levels (DirectX and C++) for more details.

Build date: 3/19/2013