Benchmarking on Windows XP
Home Edition and Professional
Updated: December 4, 2001
Microsoft Windows XP offers excellent overall performance in part because it dynamically adjusts itself based upon individual use. This article addresses some of the dynamic tuning capabilities of Windows XP, and outlines steps you should take in order to get consistent benchmark results. It provides information on how to avoid System Restore checkpoint activity during benchmark runs.
On This Page
The Windows XP operating system achieves excellent performance in part by dynamic adjustments that it makes in response to how a system is utilized. In particular, many adjustments are made when the system first runs following setup and when newly installed applications are used for the first time. These adjustments greatly influence the performance of the system. Unfortunately, the dynamic nature of Windows XP can make benchmarking more of challenge than with previous Windows versions.
The purpose of this article is to familiarize the reader with the dynamic tuning done on Windows XP, to explain why performance on Day 3 after installing the system and applications may be better than on Day 1, and to outline a testing methodology that is both representative of what real customers will experience and capable of producing consistent results from benchmark run to benchmark run.
General Concepts: The Dynamic Nature of Windows XP
Windows XP is a dynamic, self-tuning operating system, and has built-in features that support this both at setup and during continued use. As a consequence, the systems performance after having the opportunity to tune itself will be different from its performance when it boots up for the first time or when it executes a particular application for the first time.
Operating System Setup
The video-based adjustments are determined by running a variety of atomic tests after the video resolution and color depth have been automatically determined. If a specific atomic test takes too long to complete, then the corresponding visual effect is turned off by default. Microsoft added the visual-based adjustments to ensure a responsive user experience with the new user interface and visuals. The majority of systems will have all of the visuals on by default; a few systems with video driver or video adapter performance issues will be configured with the effects off by default.
The following effects are dynamically configured during setup:
Although Microsoft strongly recommends using the defaults because that is what most users will use, the configuring of visual effects settings can be done quite easily on Windows XP. To display the full list of easily configurable visual effects settings, right-click My Computer, select Properties, select the Advanced tab, and select the Settings button under Performance. The Performance Options dialog box is shown below in Figure 1.
Figure 1: Performance Options dialog box used to manually configure visual effects.
Disk Efficiency Optimizations
The files moved for more efficient layout are also kept contiguous. Windows XP does not intentionally fragment files as was done by some earlier versions of Windows.
Even though the disk layout optimization does insure some files will be defragmented, it is not a complete substitute for fully defragmenting the disk. Users should still defragment their drives regularly. The built-in Windows XP defragmentation program understands the file layout directives and will position the files if they arent already placed properly. However, it will not update the layout file with new information gathered in the last three days. Without manual intervention, the layout file, %windir%\Prefetch\Layout.ini, will only be updated once every three days. The contents of the Layout.ini file can be viewed with Notepad.
Benchmarking on a Dynamic and Self-Tuning Windows XP
Benchmarking an operating system that is attempting to improve itself over time based upon observed usage patterns can be difficult. It becomes ever more difficult to benchmark if the operating system is also attempting to create dynamic system restore points when large amounts of applications are installed. Fortunately, there is a methodology that provides consistent results, accurately reflects the performance that Windows XP systems will achieve, and is fairly easy to follow.
The basics of the methodology are:
Idle Task Scheduling: The ProcessIdleTask API
The file placement optimization, which is done no more often than once every three days, is an example of a task that is carried out when the system is deemed to be idle. System Restore and other features of Windows XP also attempt to defer some work until the system is deemed to be idle. There are also some done-once-after-setup work items that also operate under the Idle Task Scheduling mechanisms.
All of these "idle tasks" are controllable by a system API in advapi32.dll, ProcessIdleTasks. The APIs sole purpose is to allow benchmarks a simple way to force any pending idle tasks to be executed immediately, without having to wait a lengthy period of time.
The API ProcessIdleTasks can be called in one of two ways, from the command line or synchronously from a program. To call ProcessIdleTasks from the command line, use this syntax:
When called from the command line, the ProcessIdleTasks work is done in the background asynchronously. It can take 10 to 15 minutes for idle tasks to complete. Task Manager will report processes running, and the disk will likely be active during this time.
The ProcessIdleTasks API takes no arguments and returns a standard Win32 error code. The definition is:
The API itself is synchronous, so it wont return until the idle tasks have completed.
Other Testing Issues
Clean Installation Preferred
When performing a clean install, Microsoft recommends that NTFS be used and that the system be installed in a single partition on each disk. Under Windows XP, big partitions are better managed than in previous versions of Windows. Forcing installed software into several partitions on the disk necessitates longer seeks when running the system and software.
If you do choose an upgrade from Windows 2000 or Windows 9x, you may be working with a FAT32 file system. Performance will generally be better if the file system is left as it is, rather than converted to NTFS. A partition converted from FAT32 to NTFS may have to use 512-byte clusters, rather than 4096-byte or 8192-byte clusters, which can result in a higher number of fragmented files.
Installation in 64 MB Differs from Installation in 128 MB
First and Second Boots Following Installation
Windows Activation and Windows Update
Note that you can still create master images that have indefinite shelf life using standard RTM product keys (but not for MSDN keys or evaluation keys/media). This is accomplished by installing the image master with any standard retail key. The master should not be activated. As the final step prior to shutdown, run sysprep (available on the distribution media from the Support\Tools folder in Deploy.cab). This will reset the activation logic to provide indefinite storage prior to imaging and the full 30-day activation grace period for clones. Sysprep will reset the activation grace period on any master or derivative up to three times.
Because activating large numbers of machines can be undesirable, you might want to run most of your benchmarks on unactivated systems and reserve your activation tests for the final round, or in cases where you are concerned that activation might be a problem. If you must activate large numbers of machines, the draft WPA Deployment white paper available on the Windows Beta web site might provide some useful advice.
Similarly, Windows Updates should either be applied or disabled so that the system will not try to check whether updates are available in the middle of benchmark runs. Microsoft believes most consumers will apply the updates so it is best to apply the updates before testing.
Windows XP can deliver excellent performance, but because it is more sophisticated than previous versions of Windows, care must be taken to obtain valid results. Understanding the issues described in this article can help you arrive at better benchmarking practices and save time and effort
For More Information
For the latest information on Windows XP, check out the Windows XP home page.
To learn more about how Microsoft and its OEM partners are working to improve boot and resume times, see Fast Boot/Fast Resume for the Windows Platform.