File System Performance Tests for Flash (Compact 7)

Article
03/14/2014

3/12/2014

The File System Performance Test measures the performance of a file system driver (FSD) that uses the Win32 application programming interface. You can use this test to obtain throughput metrics for various file system usage scenarios on a storage device block driver that runs under the FAT, ExFAT, TexFAT, or a custom FSD. The test can also measure the performance of the Object Store file system.

To judge the performance data produced by this test, you need a basis for comparison. For instance, you might use a known theoretical throughput maximum for a specific storage device, or you might compare two data sets obtained by running this test on two different devices, FSDs, or block drivers. The following are examples of useful applications of the File System Performance Test:

* Comparing file system I/O to block driver I/O (the latter can be obtained by running the Storage Device Block Driver Performance Test).

* Comparing file system performance on storage devices from different manufacturers.

* Running a performance test every time there is a known change in any of the underlying components in the file system stack.

This test will generate a number of performance metrics. After running the test, some of these tests can be viewed visually via the GraphTool in the Windows Embedded Compact Test Kit (CTK) application. Other tests' results can be viewed in the fsperflog.xml log file that is generated. These tests were not graphed because they generate one dimensional or a single data point.

Test Name	In GraphTool	Description
Throughput Writefile	Yes	Measures file write throughput using varying buffer sizes
Throughput Readfile	Yes	Measures file read throughput using varying buffer sizes
Throughput Writefile Random	Yes	Measures the throughput while writing varying buffer sizes of data at random offsets
Throughput Readfile Random	Yes	Measures the throughput while reading varying buffer sizes of data from random offsets
Throughput Reverse Readfile	Yes	Measures the throughput while reading a file in reverse using varying buffer sizes
Throughput Overwrite	Yes	Measures the throughput to overwrite an existing file with varying buffer sizes
Create Directory Tree	Yes	Measures the directory depth that can be created per second
SetFilePointer and SetEndOfFile	No	Measures the time it takes to create a large file with SetEndOfFile
Seek Speed(read)	No	Measures how long it takes to perform a specific number of seeks followed by file reads
Seek Speed(write)	No	Measures how long it takes to perform a specific number of seeks followed by file writes
Read/Append to Fragmented File	No	Time to read and write to a large fragmented file
Copy Files(from storage to device root)	No	Time to copy a set of files from storage to device root
Copy Files(from device root to storage)	No	Time to copy a set of files from device root to storage
FindFirstFile FindNextFile	No	Time to perform FindFileFirst and FindFileNext on 50/100/200 files of the same type
FindFirstFile FindNextFile Random	No	Time to perform random FindFileFirst and FindFileNext on 50/100/200 files of the same type
Create File CREATE_NEW	No	Time to create files with different depths across different directories
Create File OPEN_EXISTING	No	First create files with different depths across different directories and measure the time to CreateFile with the OPEN_EXISTING attribute

Test Prerequisites

Your device must meet the following requirements before you run this test.

The following table shows the hardware requirement for the File System Performance Test for Flash.

Requirement	Description
Onboard flash memory	The device must have onboard available that can be used for storage.

A storage device is not required if the test is intended to measure the performance of the Object Store.

The following table shows the software requirements for the File System Performance Test for Flash.

Requirement	Description
Tux.exe	Tux Test Harness, required for executing the test.
Kato.dll	Kato logging engine, required for logging test data.
Fsperflog.dll	Test library.
ceperf.dll	Module containing functions that monitor and log performance.
perfscenario.dll	Module containing functions that monitor and log performance.

Note:When you run the File System Performance Test, the CTK temporarily copies files to the root directory of the target device. Before running the test, verify that there is at least 0.5 megabytes (MB) of free space in the root directory of the target device. If there is not sufficient space in the root directory of the target device, the test cannot run.

Subtests

The table below lists the subtests included in this test.

SubTest ID	Description
2001	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2002	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2003	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2004	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2005	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 2KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 2KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2006	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 4KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 4KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2007	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 8KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 8KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2008	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 16KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 16KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2009	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 32KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 32KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2010	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 64KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 64KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2011	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2012	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2013	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
2014	Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1MB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1MB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.
3001	Measures the speed of creating a deep nested directory tree 10 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.
3002	Measures the speed of creating a deep nested directory tree 15 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.
3003	Measures the speed of creating a deep nested directory tree 20 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.
3004	Measures the speed of creating a 500MB file by using SetFilePointer and SetEndOfFile functions. The test fails if the file cannot be created, or if either of the operations fail.
3005	Measures the seek speed by performing 1000 random seeks followed by 1 byte reads using the SetFilePointer and ReadFile functions. The test fails if the file cannot be created, or if any of the seeks or reads fail.
3006	Measures the seek speed by performing 1000 random seeks followed by 1 byte writes using the SetFilePointer and WriteFile functions. The test fails if the file cannot be created, or if any of the seeks or writes fail.
4001	Measures the I/O throughput by continuously reading and appending data to a 100MB file using the SetFilePointer, ReadFile, and WriteFile functions. This test is intended to simulate the behavior of a Digital Video Recorder (DVR) device, which often records and plays back data from the same file. The test fails if the file cannot be created, or if any of the seeks, reads, or writes fail. If the storage device has insufficient space (~300MB), the test will skip.
4002	Measures the speed of copying a folder with 100 files from the storage device to the root directory of the device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store.
4003	Measures the speed of copying a folder with 100 files from the root directory to a storage device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store.
5001	Measures the time to perform FindFileFirst and FindFileNext on same type of 50 files.
5002	Measures the time to perform FindFileFirst and FindFileNext on same type of 100 files.
5003	Measures the time to perform FindFileFirst and FindFileNext on same type of 200 files.
6001	Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 50 files
6002	Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 100 files.
6003	Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 200 files.
7001	Measures the time to CreateFile with the CREATE_NEW attribute up to 10 files across different directories and nested directories.
7002	Creates files with different depths with 10 files across different directories and nested directories. Then measures the time to CreateFile with the OPEN_EXISTING attribute.

Setting Up the Test

* The onboard flash memory must be mounted and able to read/write accessible.

* You must modify the default command line of the test to specify the profile name of the flash storage if it is not "FlashDisk." Or use other parameters to specify the existence of the flash memory volume.

* The onboard flash memory can be partitioned and formatted using the Storage Manager Control Panel Applet. This applet can be added to the image using the SYSGEN_STOREMGR_CPL. After the image has been built, the Storage manager applet can be accessed by My Device->Control Panel->Storage Manager.

Running the Test

The command line for the File System Performance Test for Flash must be modified to point to the onboard flash memory. For example, if the flash memory uses the profile name of "FlashDisk" then use the option, "-p FlashDisk".

The test will generate raw performance data in a log file named, fsperflog.xml, in the release directory.

You can modify the test by editing the command line. The following table shows the command line parameters for the File System Performance Test for Flash.

tux -o -d fsperflog -c "[-p <profile>|-r <path>|-root] [-nb -wt -flush] [-cpu] [-desc <description>]"

Command line parameter	Description
-p <profile>	Specify the storage profile to which to restrict testing, found in the registry under the HKEY_LOCAL_MACHINE\System\StorageManager\Profiles key.
-r <\path>	Specify the directory in which to run the test. If you provide this option, the test only runs in the specified directory and ignores the -p option. This flag is useful for a file system driver that does not use storage profiles such as a network file system driver. The path does not require quotation marks; for example, -r \Storage Card
-root	Specify that the test should run on the root file system. If you provide this option, the test runs in the root directory and ignores the -p and -r options.
-nb	Specify that all the test files should be opened with FILE_FLAG_NO_BUFFERING. This option can be combined with -wt.
-wt	Specify that all the test files should be opened with FILE_FLAG_WRITE_THROUGH. This option can be combined with -nb.
-flush	Include the calls to the CloseHandle and FlushFileBuffers functions in performance timing calculations.
-cpu	Include the CPU usage information in the performance log.
-desc <string>	Include a user-specified description string in the performance log.
-iosize #	The I/O buffer size for test cases (default = 512)
-filesize	The file size for test cases 2001-2014 (default = 2MB)
-rand	For tests that perform random I/O, uses TUX-provided random seed.

Verifying the Test

When the test completes running, verify that "PASS" appears in the test log for all sub-tests. A "PASS" test result indicates that the test ran successfully, but the test engineer must manually evaluate the results for each device. Investigate any warning messages that are present in the test log.

After running the test, you can view the results in two ways:

* CEPerf output log - fsperflog.xml

* GraphTool output view

In GraphTool output view, you will be able to see the graphs for tests which generate a series of data. You will have to determine if the performance numbers reported by the test meet the performance required by the device. If not, additional optimizations may be needed. The tests which are graphed are:

* Throughput Writefile

* Throughput Readfile

* Throughput Writefile Random

* Throughput Readfile Random

* Throughput Reverse Readfile

* Throughput Overwrite

* Create Directory Tree

Troubleshooting the Test

Ensure that the flash memory is available and is formatted with the specific file system that are supported by the device.

For additional platform specific issues, consult the CTK articles on the TechNet wiki.