Building a Diskless Automation Controller Using Windows XP Embedded
Alexander Wechsler and Horst Grashauser
Microsoft® Windows® XP Embedded with Service Pack 1
Choose the Right Storage Software: Flash is Not Flash
Setting Up the Development Environment
Collecting System Information for the Windows XP Embedded Image
Preparation of the Flash Disk for EWF
Adding and Configuring the EWF Components
Embedded devices nowadays have come a long way from the average 8-bit controller applied to simple switching tasks, to the fully equipped PC system, steering a complete production line.
Just recently, the mainly proprietary or specialized operating systems applied in those scenarios got competitors from the mainstream desktop and server arena. Especially in areas where flexibility, broad availability of software, and reuse of already existing communication infrastructure like Transmission Control Protocol/Internet Protocol (TCP/IP) is a benefit, mainstream operating systems can play their full deck of cards.
But this is not all, flagship operating systems like Microsoft® Windows® XP Embedded come with special embedded-enabling features, which make them, out of the box or with a little help from third parties, most appealing to a device manufacturer. With the help of those features, Windows Powered embedded devices are able to perform hard real-time tasks while still providing the richness and power of a Microsoft Win32® application programming interface (API).
Many embedded devices deny the use of a hard disk in their target environment. Be it that vibrations and dust of a rugged environment diminish the lifetime of such a component, or that just the sheer noise of the disk would spoil the user experience in a consumer product. In those cases, Windows XP Embedded provides the means to enable the use of nonvolatile memory (also known as CompactFlash) as a persistent media.
This white paper describes the approach of using CompactFlash as the storage media for a Windows XP Embedded operating system image in read-only as well as read-write mode.
CompactFlash Removable Media
CompactFlash has become a very popular storage media for all kinds of consumer products such as cameras and MP3 players. It has also become a replacement for floppy disks since it has far more capacity, for example: common CompactFlash form factors include PC Card adapters and universal serial bus (USB) sticks. Due to the very high production numbers, these kinds of removable flash cards are pretty cheap and therefore may be something you should consider using in a diskless Windows XP Embedded system.
But there are some implications if you use these flash cards with Windows XP Embedded. They will only be recognized by Windows XP Embedded as a removable media and therefore only one partition will be accessible, even if you have created more than one by using a MS-DOS boot disk and fdisk.
If this is not an issue and only one partition is intended to be used, the CF-card just behaves like a normal hard disk and can be formatted to host a file system used for the Windows XP Embedded operating system image.
If the intention is to use the Windows XP Embedded Enhanced Write Filter (EWF) to write-protect the image on the partition, this will fail, because EWF needs an additional empty partition to hold its own persistent information. This extra partition will not be recognized with the removable bit set. For more information on using EWF with CompactFlash, please see Using Compact Flash (CF) with the Enhanced Write Filter (EWF).
CompactFlash Fixed Media
The only difference between a removable flash card and a fixed flash card is one bit which is set on the card to determine its mode. There are tools available from the flash card manufacturers which allow switching this mode on and off, but they have to be acquired directly by the manufacturer in a quite lengthy process. Most often, nondisclosure agreements and licensing terms have to be signed until the tool is shipped to you.
Another way to get fixed media CompactFlash cards is to buy them, with the bit switched to "fixed" from the manufacturer. This has a clear advantage over switching the bit yourself, especially if you look at the production process. Fixed cards save one step during the device integration.
But what are the benefits of fixed flash cards? The benefit of using them with Windows XP Embedded is that they behave like a regular hard disk on your Integrated Drive Electronics (IDE) controller. Meaning, that it is easy to create and address different partitions in the system.
A common use case for embedded devices is to have a read-only system partition and a writeable data partition to persist important results or settings for applications. But be careful, the read access to the writable partition should be optimized to occur as seldom as possible to insure a long lifetime of the flash card and therefore reduce flash wear.
Flash disks are a different kind of fixed mode flash cards, which are hosted in a double CF-card size case and dock to the IDE controller by using a high speed access controller. These flash disks beat the performance of a normal CF-card by far, but offer this benefit at a higher price and larger space consumption in the embedded device.
Building an operating system image can be a very lengthy process if you are not trying to streamline the iterative cycle of building, deploying and testing in an efficient way. Here are some recommendations to achieve a proper build environment:
The Windows Embedded Studio tools should be hosted on a capable developer workstation with enough memory, at least more than 500 megabytes (MB) (1 gigabtye [GB] is not too bad). It should have PC-Card slots for the CF-cards used with an adapter or provide the means to hook a flash disk to the IDE controller.
The Windows XP Embedded database should be either located locally to provide the best performance, or on a database server to enable team development.
If possible, about three or four target devices and at least a matching number of storage media should be available for testing the freshly built images. This allows parallelizing the runs of the First Boot Agent (FBA) or other tests with the build and deployment (copying) process on the developer machine. This will shorten down the development time.
A normal build/test cycle would look like this:
- Build the image on your development machine with Target Designer
- Copy image to the CF-card/flash disk
- Move the flash disk to a test machine
- Start the test machine
- First Boot Agent Phase 1
- Automatic Restart of test machine
- First Boot Agent Phase 2
- If required, manual installation phase
- Seal/Reseal Phase
- Production Image
Of course, steps nine and ten are only applicable if all development issues are sorted out and the image is ready for production; otherwise, the whole process starts anew from the beginning after step 8.
The best and most efficient way to gather the required hardware information is to install Windows XP Professional on one of the target machines and then use the Target Analyzer Probe utility (TAP.EXE), which normally resides in the [YourDrive]:\Program Files\Windows Embedded\utilities folder.
It investigates the Windows XP registry for all hardware-related data acquired by the Windows XP setup and creates an Extensible Markup Language (XML) file called "devices.pmq," which is used to import this information into the component database using the Component Designer tool.
But Target Analyzer really collects as much information as possible. If you are building an embedded device, you may think of stripping everything out of the operating system except the most needed components to get the smallest memory footprint. In this case, it might even be that you are not using several hardware components available in your target device.
To get rid of this ballast, you can choose between two approaches. The first approach is to edit the "devices.pmq" XML file and throw out the unneeded parts manually.
<?xml version="1.0" ?> <!DOCTYPE HIB (View Source for full doctype...)> - <HIB DTDRevision="4"> - <TOOLINFO> <PLATFORM>Microsoft Windows XP Build 2600</PLATFORM> <TOOL>Target Analyzer Probe Version 2.00.807.0</TOOL> <TOOLOPTIONS>"C:\Program Files\Windows Embedded\utilities\tap.exe"</TOOLOPTIONS> <TIMESTAMP>2003-02-24, 01:23:26PM</TIMESTAMP> </TOOLINFO> - <DEVICES> - <CATEGORY Name="ACPI"> - <DEVICE ConfigFlags="0"> <DEVICEDESC>Microsoft AC Adapter</DEVICEDESC> - <HARDWAREIDS> <DEVICEID Order="1">ACPI\ACPI0003</DEVICEID> <DEVICEID Order="2">*ACPI0003</DEVICEID> </HARDWAREIDS> <COMPATIBLEIDS /> </DEVICE> - <DEVICE ConfigFlags="0"> <DEVICEDESC>Docking Station</DEVICEDESC> - <HARDWAREIDS> <DEVICEID Order="1">ACPI\DockDevice&_SB_.PCI0.PCIE.GDCK</DEVICEID> <DEVICEID Order="2">ACPI\DockDevice</DEVICEID> </HARDWAREIDS> <COMPATIBLEIDS /> </DEVICE> - <DEVICE ConfigFlags="0">
This approach can only be recommended for very experienced developers, who are aware of the dependencies between the hardware and the software components.
Another way, which is also a little bit safer, is to import the file into Component Designer and to specify the "Selector Prototype Component" as the base component for the hardware macro component.
Figure 1. Specifying the selector prototype component
This component allows you to select and deselect parts of the hardware macro component in the Target Designer IDE before each build. This provides an elegant and efficient way to test different software/hardware variations without the need of manual editing and re-importing of components and "devices.pmq."
Figure 2. Hardware macro component based on selector prototype component
After you have collected the hardware related information, you can either add software components from the Windows XP Embedded component database or create components with Component Designer. Especially when using a flash store of limited size, you should take care to add only the components you need. Therefore, do not rely solely on dependency checking in Target Designer to quickly build your image. Dependency checking always tries to build a running image by adding a lot of components to satisfy component dependencies, many of which might not be needed for your device. The only way to get the minimal system image is to strip away all additional components and to do proper testing yourself. Better yet, make sure that your specification is granular and complete enough that you know exactly what components you need to incorporate into your Windows XP Embedded operating system image.
While building the operating system image, the CF-card can be prepared by using an MS-DOS boot floppy disk on one of the test machines. The first step is pretty straightforward. Just use fdisk to create a primary partition and format it with the /s option as a system partition. The additional partitions, that needed should not be created with fdisk, but with the Disk Manager Microsoft Management Console (MMC) snap-in of Windows XP. Using EWF, there has to be an empty partition for holding the EWF master volume table of at least 32 KB.
After this the CF-card/flash disk is ready for the deployment (copying) of the generated Windows XP Embedded Image.
A special note: If it is not possible to prepare the card with Windows XP, for example, in the production environment using an MS-DOS boot disk, use fdisk with the /fprmt switch to be able to create and format a FAT32 partition.
As an example, let us assume we want to create a system with a write protected system partition and a second unprotected data partition, which will be used from our applications to store settings and other important data.
To achieve this configuration, we have created three partitions on the 256 MB CF-Card using 200 MB for the operating system image, 200 KB for the EWF partition and the rest as data partition.
Three components in Windows XP Embedded are needed to provide the full EWF functionality in a system Image:
- EWF Manager console application
- EWF Loader, either for FAT or NTFS
Enhanced Write Filter (EWF)
The main component of EWF adds the functionality plus the ability for configuration to the system image. Generally, there are two types of overlay available: Disk overlay, for IDE and small computer system interface (SCSI) disks, which protects one partition while persisting all changes in an additional overlay partition. There can be several overlays specified using disk overlays, providing even the possibility to switch back to one of several older configurations after having altered the existing one. The second option is random access memory (RAM) overlay, which is of course not persistent and therefore allows only one level of overlay.
A configuration protecting the first partition using RAM overlay could look like this:
Figure 3. EWF on CompactFlash with RAM overlay
The important settings in this window are number of protected volumes, disk number and partition number, disk type, and overlay type. Because we are using RAM overlay, we try to be as compact as possible choosing the third of the optimization options using less space and writes. The EWF partition size is not as important as long as the available space in the EWF partition is larger than the specified number.
Start EWF is not enabled, because you may want to adjust some settings manually after FBA has run. Enabling it, would mean that all changes made would be not persisted in the image but stored only in RAM. Nevertheless, by not activating EWF here, you have to take care to do this manually, later on.
EWF Manager Console Application
Exactly for executing the manual control of the EWF you need to add the EWF Manager Console application. It provides a variety of commands to enable, disable, check and view the status of the EWF. You can even influence the EWF behavior of a certain partition after boot.
Enabling the EWF on the first partition, for example, looks like this: EwfMgr c: -enable.
The EWF Manager is also used in combination with its commit command to persist changes to the originally write protected system partition (for example: EwfMgr c: -commit). Only through this feature field upgrades of the operating system image are possible.
EWF Loader NTFS/FAT
The EWF Loader is normally used in combination with hard disk based overlays, because it provides the means to write into the disk-based overlay during boot even before the operating system has loaded the disk driver. Because our sample configuration uses only a RAM based overlay and RAM is always accessible during boot, the EWF loader component is not needed and can be left out in this configuration.
After building and deploying the image, all the EWF mechanisms are configured at the end of the second FBA phase (after the automatic reboot).
Figure 4. FBA and cloning phases
At this point, a little segue into the sequence of FBA Phases, Boot Phases and Cloning seems to be appropriate. A normal Windows XP Embedded startup sequence looks like this:
- First Boot
- FBA Phase 1-8500
- Second Boot
- FBA Phase 8500-12000
- Automatic sealing of the system with the System Cloning Tool (if part of your image)
- Cloning of the system via a disk copy tool
- First start of the clone (new system security ID gets generated)
- FBA Phase 12001+
- Dedicated system image for the single target device
Within the first two FBA phases the system gets configured and all Component registrations are done. After this, the system is automatically sealed.
This means that with the next startup a new and system-unique security ID is created, which is necessary to communicate, for example, over the network. If you clone the system without the System Cloning utility, all systems will have the same security ID (identifier) and therefore it will be impossible to distinguish them in a network. The result is that network communication can not be established and the systems are not usable in this environment.
Sometimes it is a good idea to postpone the sealing of a system, to have the opportunity to do an additional manual configuration of the image. This can be achieved by setting the cmiResealPhase option in the advanced properties of the System Cloning component to 0. Then the system is not sealed automatically and left running, giving the developer the possibility to do manual changes to the image.
Figure 5. Avoiding an automatic seal
If you do this, please take into account, that for the sealing of the system after the manual configuration process the FBAReseal.exe utility has to be called to seal the system again. If this is forgotten, the system will not generate a new security ID after cloning during the initial boot and therefore cannot be used in a networked environment.
EWF Status Checks and Error Logging
Of course, it is important to find out if all system work as they are intended to do.
Therefore the first place to look for information would the FBALog.txt log file in the ..\Windows\FBA directory. A successful configuration could look like this:
ConfigureEwf() Start. Getting EWF config parameters from registry. EWF Partition Size = 0 (KBytes), Levels = 1, Volumes = 1. Protected Volume Config #0 : Disk= 0,Part= 1,DiskType= IDE,Type= RAM. Enable= Disabled, Optimize= 0, LazyWrite= N. Found 2 Hard Disks. Searching for El Torito disk. Disk0 signature = 0x499602D2. Disk1 signature = 0x4F544C45. Disk1 is an ElTorito disk. Disk #0 layeout info: PRIMARY partition,start=0x00000001966dfa00, len=0x0000000025429800, type= 6 Created EWF partition on Disk = 1, partition = 1,size = 0x0000000000007e00 . Saving EWF configuration to registry: Protected Volume ArcName = multi(0)disk(0)rdisk(0)partition(1). ewfOpen. EWF Volume Config on Disk#1, Partition#1: Segments = 0, Max Volumes = 1, Max Levels = 1 ewfAdd. Protected Volume Config on Disk0\Partition1 : Type = RAM, State= DISABLED. ewfClose ConfigureEwf() End, status = 0x0
In FBALog.txt, pay attention to the status bit at the end. If it is not 0, something has gone wrong and you would have to look for additional information in the EWF section further up in the log file.
In addition, one could also try to start EWF with EWF manager and check its status.
One should see a result similar to the following:
Ewfmgr Volume Size 681574400 Segments 8192 Segment Size 249856 Free segments 8192 Max Levels 1 Max Protected Volumes 1 Protected Volumes 1 Overlay volume percent full 0.00 Protected volumes Arc Path "\Device\HarddiskVolume1"
To check the state of the protected volume:
Ewfmgr C: Protected Volume Configuration Type RAM State DISABLED Boot Command NO_CMD Param1 0 Param2 0 Persistent Data "" Volume ID D2 02 96 49 00 0E 59 96 02 00 00 00 00 00 00 00 Device Name "\Device\HarddiskVolume1" Max Levels 1 Clump Size 512 Current Level 1 Disk space used for data 0 bytes Disk space used for mapping 0 bytes Memory used for mapping 0 bytes
During development, FBA may reboot endlessly after the normal one-time boot. This behavior is caused by an old EWF volume table in the EWF partition, which FBA is not able to delete.
This sounds pretty bad up front, but there is help available. The created image can be saved by using the etprep.exe utility with its /all option (to be found next to tap.exe in the utilities folder), which deletes the old entries in the EWF partition. This will stop the endless system reboots.
Etprep is the only tool that enables you to do this. Other utilities like fdisk or partition magic cannot read the partition table EWF produced and therefore are not able to delete it.
If EWF or other applications do not start or behave strangely in other ways, this might be related to missing application components or configuration settings.
There are two tools available from Sysinternals, which help you out here: Filemon and Regmon.
Filemon hooks into the Filesystem driver, to provide a detailed overview on what is going on in the system, while Regmon does the same with the registry.
You will be surprised what is going on even on a system without user interaction. Both monitoring tools provide rich filtering options that help to find the way through the richness of activities.
Figure 6. Filemon in action
Figure 7. Regmon in action
Using Windows XP Embedded in combination with the EWF on CompactFlash enables a very flexible and robust system design without having a hard disk in the system. The modular building block approach of Windows XP Embedded, along with the closed tool chain provided, guarantee high development efficiency and thus a shorter time to market.
The key to success using Windows XP Embedded in combination with the Enhanced Write Filter is to circumvent the limitations of a removable flash card by using cards with the removable bit set to "fixed."
In addition, it is also very important to understand the mechanisms of the First Boot Agent interacting with the system cloning tool. By having the right procedures, sequence of installation and configuration in place, the quality and reliability of the production image can be assured. Of course, one also should not forget to properly test in the intended deployment environment.