Cloning the Microsoft Public Common Code

Article
08/28/2006

Mike Hall
Microsoft Corporation

Steve Maillet
Entelechy Consulting

September 24, 2003

Summary: Shows you how to make changes to the public source that ships with Microsoft Windows CE .NET 4.2. (9 printed pages)

This month we were going to write about building headless device images for Microsoft® Windows® CE.NET, building on last month's article, which described how to create a custom platform wizard for Windows CE .NET. However, there has been a flurry of issues in the newsgroups surrounding the use of the common code for Windows CE .NET, so it seemed only right to address this ahead of the headless article. So this month we will focus on modifying the common code that ships with Windows CE .NET.

Windows CE .NET ships with "shared source." This lives in the Private folder for a Windows CE installation. The product also ships with a ton of public or common source, which can be freely modified and shipped as part of your device. Directly modifying the common source sounds like a great idea, but it can also create a number of issues. In fact, the first rule of modifying the common source is...

"Don't Alter the Common Source!"

This needs some explanation. You can of course easily modify the common source in it's default installation folder, but there are a number of reasons why you should copy the source to a new folder and make changes on the copied code. Indeed, a number of developers are getting caught unaware of the problems when directly changing the common code. In this article we'll discuss these issues and cover the methods to avoid them.

**Note **This article does not cover how to build or modify the private shared source code, although this is discussed in the MSDN article, Building a Platform Using Modified Shared Source Code.

Before we get into how to create a clone of the common code, let's take a look at some of the reasons you might want to change the public common code:

You need to customize an x86-based OEM adaptation layer (OAL) for your specific system needs (like using 128 MB of memory). This is detailed in the article, How to Customize Memory on a CEPC.
You need to make an optimization to the CPU Support Package libraries for a particular CPU/ASSP implementation.
You have a device that is similar to one for which there is a standard driver; however, the hardware team configured it in such a way that the existing driver doesn't work.
You have a device that uses the next version of some chip for which a standard driver exists, and you want to take advantage of the new chip's functionality.
You need to customize the user experience for your device.

These are just a few of the possible reasons to change the common source. Certainly other reasons exist. So why not just change the code in place and get on with it? (Seems like a reasonable question.)

Well, it turns out—after working with a number of customers that have wanted to make changes to the common code—that we find there are a number of reasons not to. Here are some examples:

The common code applies to all devices you build and you may not want your changes applied to every device.
Microsoft can update the common code through a QFE (Quick Fix Engineering). (QFEs typically provide common code updates to the binaries and provide instructions on updating the source code—so if you aren't careful they can get out of sync.)
Changing the common code complicates automated builds and configuration management. (How do you manage the changes Microsoft makes and the changes you make?)
It may be easy for you to remember that you modified something in the public code to solve a problem in a pinch, but the next developer that takes a look at the source won't expect the changes and will need to spend extra time and effort trying to find out what you did.

These are just a few of the reasons not to modify the source in place. So why would anyone modify the common code directly if it causes so much pain? Well that's because it's not particularly obvious how to deal with the real-world requirements we listed earlier without changing the common code. The basic idea is to make a copy locally and build it there. However, that is a good deal easier said then done. To accomplish this requires an understanding of the Windows CE .NET build process. (See our earlier article: Windows CE: Inside the Build System.) This is further complicated by the fact that the actual solution depends on the type and location of the code you are modifying. This article is focused on reducing the pain of setting up copies of the common code in your own folders, so you can modify and control them independently of any QFE or other changes, eliminating a whole set of issues that might cause you trouble later.

OALs and the CSPs

The most common case of needing to modify the Microsoft-provided common code is in the CPU Support Packages (CSP). These are a collection of libraries commonly used in OALs for a particular CPU family. This sounds like a great idea. It's nice to have common code in a library so everyone doesn't have to write their own version. Unfortunately, reality isn't so in tune with the ideal. Silicon vendors make mistakes and chips have Errata lists. Sometimes the common code needs adjustment for a particular CPU. Furthermore it is sometimes necessary to modify the CSP code on a per BSP basis.

This section explains the steps using the x86 CPU as a sample, since that is the most common one needing modification. Keep in mind the basic steps and concepts apply to all of the CSPs and BSPs. These steps assume you have cloned the reference BSP (CEPC in this case) to your own platform <YourBSP>. (See our earlier article Building a CEPC Reference Board.)

(Thanks go to Michael Verhagen who posted an early version of these steps to the Microsoft newsgroups.)

Copy the directory OAL and INC from _WINCEROOT\PUBLIC\COMMON\OAK\CSP\I486 to \WINCE410\PLATFORM\<YourBSP>\KERNEL. (There will now be a HAL folder and an OAL folder there. The HAL folder contains your original OAL code and the OAL folder contains the copied common code.)

Modify the SOURCES file in \WINCE410\PLATFORM\<YourBSP>\OAL to generate the i486 OAL to your local BSP tree by setting the RELEASETYPE to PLATFORM. Otherwise any other x86 platforms you create may pick up the public library from the wrong BSP. Since you are modifying this for a particular system, you don't really want it to apply to all of them. (That's why we're cloning it in the first place.)

!if "$(_TGTCPUFAMILY)" != "x86"
SKIPBUILD=1
!endif
WINCEOEM=1
RELEASETYPE=PLATFORM
TARGETNAME=i486oal
TARGETTYPE=LIBRARY
TARGETLIBS= \
    $(_COMMONSDKROOT)\lib\$(_CPUINDPATH)\coredll.lib\
INCLUDES=..\inc
CDEFINES=$(CDEFINES) -Zi
SOURCES= \
     debug.c  \
     isapnp.c \
     rtc.c  \
    timer.c  \
    fwpc.c  \
    cfwpc.c  \
    mdppfs.c \
    pci.c  \
    power.c
X86_SOURCES= \
    oeminit.asm

Modify the DIRS file in \WINCE410\PLATFORM\<YourBSP>\KERNEL to add the new OAL folder:
```
 DIRS= hal  \
      oal  \
      Buildexe  
```
**Note **The order of the directories listed is important. The DIRS file lists which directories are built first. BUILDEXE needs the HAL and OAL libraries, which is why these are built first.
Modify the sources files in _WINCEROOT\PLATFORM\<YourBSP>\KERNEL\BUILDEXE, directories KERN, KERNKITL and KERNKITLPROF (Modify all three of them.)

Modify the following line:

$(_COMMONOAKROOT)\lib\$(_CPUINDPATH)\i486oal.lib

To:

$(_TARGETPLATROOT)\lib\$(_CPUINDPATH)\i486oal.lib

And make sure it's the last line on the TARGETLIBS directive. This tells the build system to use the BSPs local version of the i486 OAL code, instead of the public common one.

The sources file in KERN for example, should look like this:

SYNCHRONIZE_DRAIN=1
RELEASETYPE=PLATFORM
EXEENTRY=StartUp
EXEBASE=0x80200000
TARGETTYPE=PROGRAM
TARGETNAME=kern

INCLUDES=$(_PUBLICROOT)\common\oak\csp\x86\loadcepc\inc; \
         $(_PUBLICROOT)\common\oak\csp\i486\inc

TARGETLIBS =    \
    $(_COMMONOAKROOT)\lib\$(_CPUDEPPATH)\nk.lib \
    $(_TARGETPLATROOT)\lib\$(_CPUINDPATH)\hal.lib \
    $(_COMMONOAKROOT)\lib\$(_CPUINDPATH)\PCIreg.lib \
    $(_COMMONOAKROOT)\lib\$(_CPUDEPPATH)\loadauth.lib \
    $(_COMMONOAKROOT)\lib\$(_CPUINDPATH)\fulllibc.lib \
    $(_TARGETPLATROOT)\lib\$(_CPUINDPATH)\i486oal.lib

!IFNDEF KERNELNOSHAREETH
CDEFINES=$(CDEFINES) -DIMGSHAREETH
TARGETLIBS=$(TARGETLIBS)
$(_COMMONOAKROOT)\lib\$(_CPUDEPPATH)\vbridgestub.lib
!ENDIF

LDEFINES=-subsystem:native /DEBUG /DEBUGTYPE:CV /FIXED:NO

SOURCES=kitlstub.c

You can now rebuild your system, and any changes made in the local copy of the OAL are now incorporated into your new build.

Device Drivers

After the OAL and CSPs, the next common code files typically changed are drivers. To modify a driver you first copy the driver code and sources file and update it to build locally. This gets a bit more complex with most of the drivers, as they are often built from multiple libraries and typically build a library from the sources file. So, if the SOURCES file generates a library, how does the system build the actual DLL? Well that's where things get interesting.

Due to the configurable nature of the operating system, the build system waits until all the system libraries are built before it links everything together. The actual DLL is generated from a MAKEFILE in the public branch. The sysgen phase of the build will run the makefile in %_WINCEROOT%\PUBLIC\\<projname>\CESYSGEN\ This MAKEFILE generates the DLL from the various libraries it needs. Since all we are after is one driver (or any other single DLL/EXE for that matter), we need to somehow extract the information to link the customized copy independent of the MAKEFILE. Fortunately, with V4.2, Microsoft created a set of tools to assist with this process. Unfortunately it didn't make it into the documentation. The following is extracted from the README.HTM that is provided with Platform Builder and explains how to use the tools:

Windows CE .NET 4.2 introduces two utilities, Sysgen_capture.bat and Nmake_capture.exe, to assist OEMs who want to rebuild modified public drivers in their platform directories. Previously, this was difficult to do, because the Sysgen process links using environment variables dynamically generated within the build system. These utilities capture and reveal the internal dependency information used within the build system. OEMs can use the information captured by these utilities when building drivers within their platform directories.

The following steps use the serial port driver for 16550 UARTs, Com16550.dll, as an example to show how to use Sysgen_capture.bat to create Sources files for driver modules normally generated by the build system during the Sysgen phase. Use the general process to expedite copying public code to your platform, so you can make custom modifications.

**Note **Sysgen_capture.bat only works on Sysgen targets that are dynamically linked libraries (DLLs) or executables.

Create two subdirectories for the driver within your platform directory. One is for a library corresponding to your driver, and the other is for your driver's binary file (.dll or .exe).

For example, to customize Com16550.dll, create a subdirectory named %_TARGETPLATROOT%\Com16550\Src for the library, and %_TARGETPLATROOT%\Com16550\DLL for the binary file.

**Note **If you were customizing an executable like Etcha.exe, then you would create %_TARGETPLATROOT%\Etcha\Src and %_TARGETPLATROOT%\Etcha\EXE.

Verify that the library name for your driver module is different from the name for your driver's binary file. In this example, Com16550.dll is built by linking Com16550_lib.lib. Most DLL Sysgen targets use the *_lib.lib convention to differentiate the library file and binary file names. Otherwise, the driver's import library has the same name as the static library containing its code. (Not a good thing as it would be wrong in most cases.)
Copy the public directory containing the module you want to build into the Src directory you created. In this example, you would copy the files from Public\Common\OAK\Drivers\Serial\Com16550\*.*.
Add the following line to the file Com16550\Src\Sources:
```
RELEASETYPE=PLATFORM
```
If a RELEASETYPE line already exists in the Sources file, then change its value to PLATFORM. This tells the build system that output files should go into the platform's Lib or target directories.
Verify that the following line appears in Com16550\Src\Sources:
```
WINCEOEM=1
```
This tells the build system to include portions of header files only available to OEMs building platforms. (Note that you can also set this for all of your drivers in sources.cmn in the BSPs root directory.)
Build the library in Com16550\Src. If there are any problems, resolve them before proceeding. When successful, the build produces Com16550_lib.lib in the appropriate subdirectory of %_TARGETPLATROOT%\lib.
Change to the directory for your driver's binary file, for example, Com16550\DLL.
Run Sysgen_capture with the usual Sysgen parameters, for example:
```
Sysgen_capture com16550
```
This should create one or more files, one of which is a Sources file with the name of the driver as the file extension, for example, Sources.com16550. The Sysgen_capture.bat utility creates one such file for each .dll or .exe target created while running your Sysgen command.
Rename the Sources file for your driver as Sources. For example, remove the extension .com16550 from Sources.com16550.
Copy one of the boilerplate Makefiles into your Src directory. For example, copy com16550\Src\Makefile into com16550\DLL. At a minimum, you need a Sources file and a Makefile to run Build.exe.
Run Build.exe. This should create the same binary as the standard Sysgen command, but in your platform's target directory, which is a subdirectory of %_TARGETPLATROOT%\target. You can check Build.log to determine where the binary was created.

**Note **This version of the driver is using the standard version of Com16550_lib.lib, not the one you created in Com16550\src.
Edit the Com16550\src\sources file. Remove the TARGETLIBS (or SOURCELIBS) entry for Com16550_lib.lib and move it to SOURCELIBS, changing the path as appropriate. You may need to replace $(_PUBLICROOT)\common\oak with $(_TARGETPLATROOT).
Run Build.exe again in the directory for your driver's binary file. For example, Com16550\DLL. You may need to run it with the –c option to delete all of the object files. It should build the target binary using your version of Com16550_lib.lib. You can confirm this by looking in Build.log.
Add a Dirs file to the Com16550 directory that lists the Src and DLL (or EXE) directories, in that order. You can then build your modified version of Com16550\Src and link an executable with one build command.

Run Build.exe with the –c option in the %_PROJECTROOT%\CustomDriver directory and confirm that the contents of both the Src and DLL (or EXE) subdirectories are built, and that the driver links.

The Sysgen_capture utility also works on operating system components that are not drivers. The following table shows commands for other operating system components.

Command	Result
Sysgen_capture pm	Creates a Sources file for the Power Manager.
Sysgen_capture etcha	Creates a Sources file for Etcha.exe, a touch driver test program.
Sysgen_capture –p wceshellfe cplmain	Creates a Sources file for the control panel.

To use this utility effectively, you can consult the documentation for Sysgen.bat.

Well, there you have it! Everything you need to create a build of custom copies of a single module or set of modules in the public COMMON code.

Entire Public Branches

There is the possibility that just a single module isn't enough. On a few rare occasions, you might need to do a little more. Perhaps you are providing your own custom thin client and wish to have a customized version of the shell with a number of related modules that go along with it. You wouldn't really want to perform the previous solution on every module for that. Instead you can make a copy of the entire directory and give it a new name like MYRDP. The details of making this work depend on the individual folder you are trying to copy, but basically its a copy and build process.

The way the COMMON versions are built is through an environment variable called _DEPTREE, which is set up in WINCE.BAT. This batch file will set up _DEPTREE to include each of common operating system folders, and will scan all of the folders directly under _WINCEROOT\public looking for a .CEC file. (The contents of the CEC file don't actually matter; it can even be empty.) It adds any folders with a .CEC to the _DEPTREE. So the process of "cloning" an entire public folder is mostly one of copying the folder and creating a blank CEC file to get your folder as an automatic dependency.

In most cases this is probably all you will need to do to get a customized folder to make your own modifications in. You may need to handle any of the build or data files that might have hard-coded paths referring to the original location. The simplest method of knocking those out is to temporarily rename the original folder to something else and run the build and see what falls out. Once you have your version building correctly, you can rename the original back again.

Summary

So there it is, in one place, all of the steps to make and use clones. (And not a word about DNA or double helixes nor even a mention of "clone wars" or "Use the SOURCES Luke" either! It wasn't easy you know!) Modifying the common code in place can lead to headaches, product delays, and the end of the world as we know it. All of which are generally considered... well, "bad." Cloning the code allows you to make changes as appropriate without fear of the hidden pitfalls of modifying the Microsoft public common code directly in place, leaving peace, love, and happiness for all.

Get Embedded

Mike Hall is a Product Manager in the Microsoft Embedded and Appliance Platform Group (EAPG). Mike has been working with Windows CE since 1996—in developer support, Embedded System Engineering, and the Embedded product group. When not at the office, Mike can be found with his family, working on Skunk projects, or riding a Honda ST1100.

Steve Maillet is the Founder and Senior Consultant for Entelechy Consulting. Steve has provided training and has developed Windows CE solutions for clients since 1997, when CE was first introduced. Steve is a frequent contributor to the Microsoft Windows CE development newsgroups. When he's not at his computer burning up the keys, Steve can be found jumping out of airplanes at the nearest drop zone.