Benchmarking on Windows XP

Home Edition and Professional

Updated: December 4, 2001

Microsoft Windows XP offers excellent overall performance in part because it dynamically adjusts itself based upon individual use. This article addresses some of the dynamic tuning capabilities of Windows XP, and outlines steps you should take in order to get consistent benchmark results. It provides information on how to avoid System Restore checkpoint activity during benchmark runs.

On This Page

  Introduction
  General Concepts: The Dynamic Nature of Windows XP
  Benchmarking on a Dynamic and Self-Tuning Windows XP
  Idle Task Scheduling: The ProcessIdleTask API
  Other Testing Issues
  Summary
  For More Information

Introduction

The Windows XP operating system achieves excellent performance in part by dynamic adjustments that it makes in response to how a system is utilized. In particular, many adjustments are made when the system first runs following setup and when newly installed applications are used for the first time. These adjustments greatly influence the performance of the system. Unfortunately, the dynamic nature of Windows XP can make benchmarking more of challenge than with previous Windows versions.

The purpose of this article is to familiarize the reader with the dynamic tuning done on Windows XP, to explain why performance on Day 3 after installing the system and applications may be better than on Day 1, and to outline a testing methodology that is both representative of what real customers will experience and capable of producing consistent results from benchmark run to benchmark run.

 Top of page

General Concepts: The Dynamic Nature of Windows XP

Windows XP is a dynamic, self-tuning operating system, and has built-in features that support this both at setup and during continued use. As a consequence, the systems performance after having the opportunity to tune itself will be different from its performance when it boots up for the first time or when it executes a particular application for the first time.

Operating System Setup
There are two types of dynamic adjustments made during the setup of Windows XP: RAM-based adjustments and video-based adjustments. The RAM-based adjustments are simple; if the system has 64 MB of physical memory during setup, then:

• No background bit map will used by default

• Fast user switching will be off by default

The video-based adjustments are determined by running a variety of atomic tests after the video resolution and color depth have been automatically determined. If a specific atomic test takes too long to complete, then the corresponding visual effect is turned off by default. Microsoft added the visual-based adjustments to ensure a responsive user experience with the new user interface and visuals. The majority of systems will have all of the visuals on by default; a few systems with video driver or video adapter performance issues will be configured with the effects off by default.

The following effects are dynamically configured during setup:

• Slide taskbar buttons

• Use drop shadows for icon labels on the desktop

• Smooth edges of screen fonts

• Fade or slide menus into view

• Fade out menu items after clicking

• Fade or slide ToolTips into view

• Show shadows under menus

• Show translucent selection rectangle

• These visual effects and others can have dramatic impact on benchmark scores.
  

Although Microsoft strongly recommends using the defaults because that is what most users will use, the configuring of visual effects settings can be done quite easily on Windows XP. To display the full list of easily configurable visual effects settings, right-click My Computer, select Properties, select the Advanced tab, and select the Settings button under Performance. The Performance Options dialog box is shown below in Figure 1.

Figure 1: Performance Options dialog box used to manually configure visual effects.

Disk Efficiency Optimizations
The physical placement, or layout, of files on the disk can have a considerable effect on performance, up to 10% for normal use. Windows XP observes file usage patterns as the system is used. If deemed necessary, Windows XP will adjust the file layout at three day intervals. By placing files that are referenced together near each other on the disk, and towards the more dense outer edge of the disk, seek distances are reduced which results in shorter seek times and improved performance. The performance benefit of placing files becomes increasing important as the size of the disk increases.

The files moved for more efficient layout are also kept contiguous. Windows XP does not intentionally fragment files as was done by some earlier versions of Windows.

Even though the disk layout optimization does insure some files will be defragmented, it is not a complete substitute for fully defragmenting the disk. Users should still defragment their drives regularly. The built-in Windows XP defragmentation program understands the file layout directives and will position the files if they arent already placed properly. However, it will not update the layout file with new information gathered in the last three days. Without manual intervention, the layout file, %windir%\Prefetch\Layout.ini, will only be updated once every three days. The contents of the Layout.ini file can be viewed with Notepad.

Boot Prefetching
Windows XP speeds up system boot by observing the code and data needed each time the system is booted and prefetching the necessary file contents early in the boot process. This prefetching is not done until the third boot of the system, when sufficient information is available to make the prefetching most effective. The files observed during system boot will be used in the disk layout process mentioned above. Additional information on the prefetcher and the Windows XP boot process can be found at Fast Boot/Fast Resume for the Windows Platform.

Application-Launch Prefetching
Windows XP also uses prefetching when launching applications. The files and the contents of the files accessed by each new process are observed and recorded. No prefetching can be done for the first launch of an application, so first launches are often considerably slower than subsequent launches. About 85% to 90% of the improvement is realized after just one launch of an application, with the remaining speed improvement coming after the system has had an opportunity to adjust the disk layout with information specific to this application.

 Top of page

Benchmarking on a Dynamic and Self-Tuning Windows XP

Benchmarking an operating system that is attempting to improve itself over time based upon observed usage patterns can be difficult. It becomes ever more difficult to benchmark if the operating system is also attempting to create dynamic system restore points when large amounts of applications are installed. Fortunately, there is a methodology that provides consistent results, accurately reflects the performance that Windows XP systems will achieve, and is fairly easy to follow.

The basics of the methodology are:

• Install
  Install the OS and benchmark as normal.

• Use
  Run the benchmark one or more times.
  OR
  Use the applications in the benchmark as real users would.

• Let Idle Tasks Run
  Leave the system for a bit after using it, (up to three days for disk layout optimizations).
  OR
  Call the ProcessIdleTask API from advapi32.dll.

• Test
  Run the benchmark as usual.

 Top of page

Idle Task Scheduling: The ProcessIdleTask API

The file placement optimization, which is done no more often than once every three days, is an example of a task that is carried out when the system is deemed to be idle. System Restore and other features of Windows XP also attempt to defer some work until the system is deemed to be idle. There are also some done-once-after-setup work items that also operate under the Idle Task Scheduling mechanisms.

All of these "idle tasks" are controllable by a system API in advapi32.dll, ProcessIdleTasks. The APIs sole purpose is to allow benchmarks a simple way to force any pending idle tasks to be executed immediately, without having to wait a lengthy period of time.

The API ProcessIdleTasks can be called in one of two ways, from the command line or synchronously from a program. To call ProcessIdleTasks from the command line, use this syntax:

Rundll32.exe advapi32.dll,ProcessIdleTasks

When called from the command line, the ProcessIdleTasks work is done in the background asynchronously. It can take 10 to 15 minutes for idle tasks to complete. Task Manager will report processes running, and the disk will likely be active during this time.

The ProcessIdleTasks API takes no arguments and returns a standard Win32 error code. The definition is:

DWORD















ProcessIdleTasks()

The API itself is synchronous, so it wont return until the idle tasks have completed.

 Top of page

Other Testing Issues

Clean Installation Preferred
When benchmarking Windows XP, Microsoft strongly recommends a clean installation using NTFS. There are several reasons why performance for a clean installation will tend to be superior to that for upgraded systems. An upgraded system will constrain the placement of files and file system data. The old disk format may not use an optimal file system cluster size. In a clean installation, the placement of file system data on the disk and the internal organization of that data can be optimized, resulting in a smaller system footprint and fewer and faster I/Os when using the system.

When performing a clean install, Microsoft recommends that NTFS be used and that the system be installed in a single partition on each disk. Under Windows XP, big partitions are better managed than in previous versions of Windows. Forcing installed software into several partitions on the disk necessitates longer seeks when running the system and software.

If you do choose an upgrade from Windows 2000 or Windows 9x, you may be working with a FAT32 file system. Performance will generally be better if the file system is left as it is, rather than converted to NTFS. A partition converted from FAT32 to NTFS may have to use 512-byte clusters, rather than 4096-byte or 8192-byte clusters, which can result in a higher number of fragmented files.

Installation in 64 MB Differs from Installation in 128 MB
Windows XP determines at setup time which, if any, operating system features will be turned off by default. In 64 MB there will be no background bitmap on the desktop and Fast User Switching will be disabled by default. Thus, an installation in 128 MB or more which is then restricted to using 64 MB of memory (for example by setting the MAXMEM option in boot.ini) will behave differently than would a proper installation in 64 MB.

First and Second Boots Following Installation
As noted in the previous section, Windows XP fast boot only begins to operate on the third boot following installation. For the first boot there is simply no information available that would indicate what code or data needs to be prefetched. The first boot, moreover, must perform a large number of initialization and other one-time-only tasks. Consequently, using information about this boot as a template for subsequent boots would be a mistake. It is this second boot, therefore, that is observed and information about this boot is used for subsequent prefetching and disk layout optimizations. By the third boot, appropriate information is available that will greatly improve boot speed. Microsoft has found that about 90% of the speed improvement in boot will occur for the third boot, with the remaining 10% coming after the system has had a chance to lay out the files to improve disk efficiency.

Windows Activation and Windows Update
Microsoft recommends that you activate your Windows XP installation before using it to measure performance, because the activation detection will impose a slight cost on the system at some times and no cost at other times. The variability is the issue. Also, because users need to activate within a short period of time after installation, the vast majority of system use will be done after activation, which makes measurements taken after activation more realistic.

Note that you can still create master images that have indefinite shelf life using standard RTM product keys (but not for MSDN keys or evaluation keys/media). This is accomplished by installing the image master with any standard retail key. The master should not be activated. As the final step prior to shutdown, run sysprep (available on the distribution media from the Support\Tools folder in Deploy.cab). This will reset the activation logic to provide indefinite storage prior to imaging and the full 30-day activation grace period for clones. Sysprep will reset the activation grace period on any master or derivative up to three times.

Because activating large numbers of machines can be undesirable, you might want to run most of your benchmarks on unactivated systems and reserve your activation tests for the final round, or in cases where you are concerned that activation might be a problem. If you must activate large numbers of machines, the draft WPA Deployment white paper available on the Windows Beta web site might provide some useful advice.

Similarly, Windows Updates should either be applied or disabled so that the system will not try to check whether updates are available in the middle of benchmark runs. Microsoft believes most consumers will apply the updates so it is best to apply the updates before testing.

Video Resolution
The video resolution chosen for the system will have a strong impact upon the sizes of bitmaps and other data structure in the kernel. If the graphics subsystem can support it, the default pixel color depth for Windows XP is 32 bits per pixel. If you are comparing Windows XP with a different version of Windows, it is important to ensure that the color depth for the two systems is matched. It is also best to try to use the same drivers for both, using the most recent versions of drivers from the web, or to take the default, inbox drivers for each system and compare default to default.

 Top of page

Summary

Windows XP can deliver excellent performance, but because it is more sophisticated than previous versions of Windows, care must be taken to obtain valid results. Understanding the issues described in this article can help you arrive at better benchmarking practices and save time and effort

 Top of page

For More Information

For the latest information on Windows XP, check out the Windows XP home page.

To learn more about how Microsoft and its OEM partners are working to improve boot and resume times, see Fast Boot/Fast Resume for the Windows Platform.

 Top of page