Cats, Toast, and Interrupts on Windows CE .NET


Mike Hall
Microsoft Corporation

Steve Maillet
Entelechy Consulting

January 20, 2004

Summary: Examines the interrupt architecture of Windows CE .NET 4.2, and shows how to implement interrupt support on Windows CE .NET. (8 printed pages)

We're back with the first Microsoft® Windows® CE .NET article of 2004. The holidays are a great time to relax with friends and family and also to spend some time scanning the Windows CE .NET newsgroups for interesting questions or ideas for this column. This time of year is also a great time for people to ponder some of life's more interesting questions: Why does toast always land butter side down? Why do cats always land on their feet? If you tied a piece of buttered toast to a cat would the combination spin for ever, never being able to decide whether to land butter or cat side down? From scanning the newsgroups over the last couple of weeks, it looks like a number of you have questions about the interrupt architecture on Windows CE .NET. This is a fairly common question, and now is as good a time as any to describe how this works.

Hardware-generated interrupts typically originate from I/O devices that must notify the operating system when they need service. Pointing devices, printers, keyboards, disk drives, and network cards are all typically interrupt-driven devices. Interrupt processing is broken into two components: a kernel-mode interrupt service routine (ISR) and a user-mode interrupt service thread (IST). ISRs are generally small, very fast pieces of code. Since these are running in kernel mode, the ISR doesn't have access to any of the higher-level operating system APIs. Their sole job is to return a logical interrupt identifier to the kernel. Then the kernel sets an interrupt event on which the IST is waiting.

The first thing to note is that the interrupt architecture and driver model of Windows CE .NET is completely different from the architecture and driver model of Windows XP (although Windows CE .NET is NDIS driver-source compatible with the desktop). This is deliberate. The Windows CE .NET driver/architecture is very flat, and is one of the reasons why Windows CE .NET is a real-time operating system out of the box. (This is not to be confused with real-time communications (RTC), which is supported by both Windows CE .NET and Windows XP Embedded.) The Windows XP driver model provides a mechanism for queuing interrupts. Interrupts being queued rather than handled immediately gives an unbounded response time to hardware interrupts, and therefore makes for non-deterministic behavior on the device. Note that Windows XP Embedded can be a real-time device, though this does require third-party extensions.

Windows CE Interrupt Architecture

The following diagram illustrates the interrupt-handling architecture of Windows CE.NET:

Figure 1. Interrupt architecture for Windows CE .NET

  1. The hardware generates an interrupt.
  2. The kernel receives the interrupt and dispatches to the ISR (Interrupt Service Routine).
  3. The ISR determines the source of the interrupt if necessary, and disables the interrupt line for this IRQ at the interrupt controller. (This is typical, but not required; clearing the interrupt at the device is a valid option especially with shared interrupts.) The ISR then returns a SYSINTR value for the interrupt to the kernel.
  4. The kernel looks up an event for the SYSINTR and, if found, sets that event for any waiting threads. The scheduler eventually schedules the IST (Interrupt Service Thread) waiting on the event. The length of time before the IST thread is scheduled depends on the priority of the thread and other threads running in the system. Typically, Interrupt Service Threads are high priority threads, so there is minimal latency (Note that all threads within the Windows CE operating system are created with a default thread priority of 251). (See our discussion of real-time systems for more information on some timing tools and Real time techniques.)
  5. The IST does what it needs to do with the IRQ disabled (usually using CEDDK routines to access the hardware in a platform-independent manner). The IST should do as little as possible with the interrupts off, as other peripherals may have pending interrupts on a shared IRQ.
  6. The IST calls InterruptDone( ) to re-enable the interrupt at the controller. The IST then does whatever it needs to do to handle or process the interrupt and goes back to wait for the event.

Typical IST (Interrupt Service Thread) Start

When you think of Interrupt Service Threads (IST), you might typically think of the thread being part of a device driver, which is usually loaded by the device-driver manager, Device.exe. This doesn't necessarily need to be the case. The IST is simply a thread that's waiting on an event handle, which could be a device driver or a thread running as part of an application. We will take a look at how we create the event and register our IST shortly.

A driver will normally have an Interrupt Service Thread (IST) constantly waiting for the interrupt event to occur. The IST is normally a loop that waits for the event, processes the interrupt when the event occurs, and goes back to wait for another interrupt. However, it is important for well-behaved installable drivers to allow for unloading the driver, so you must implement the IST in such a way as to allow the driver to stop it cleanly. We'll get to the details of this in a bit. First, though, we need to actually start the thread.

The IST will need some information to operate correctly. Typically, you create a data structure to pass to the thread as a parameter to CreateThread that becomes the parameter to the thread function. It is a good idea to pass this data to the thread instead of using a global variable, as it's possible for a driver to handle more than one physical device of the same type. (For example, a system could have four 16550 serial ports but only needs one actual serial driver.)

typedef struct ISTData   // Declare the Structure to pass to IST
   HANDLE hThread;       // IST Handle
   DWORD sysIntr;        // Logical ID
   HANDLE hEvent;        // handle to the IST event 
   volatile BOOL abort;  // flag to test to exit the IST
   DWORD Priority; 

ISTData g_DeviceISTData;

// Create event to link to IST 
g_DeviceISTData.hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);

g_DeviceISTData.sysIntr = MySysIntrFromRegistry;
g_DeviceISTData.Priority = MyISTPriorityFromRegistry;

// start the thread
g_DeviceISTData.hThread = CreateThread(NULL,0, &DeviceIST, &g_DeviceISTData, 0, NULL);

The driver should read the priority and SYSINTR values from the registry. Hard coding these values makes the driver less portable to other platforms or configurations. The system provides the DDKReg_GetIsrInfo() to read all the standard ISR registry entries in one call.

Typical IST

Typically, an IST will loop waiting for interrupt events it should run at a higher priority to minimize the latency in handling the interrupt. Note that all threads created in Windows CE .NET are created at a default thread priority of 251 (0 being the highest priority, 255 being the lowest priority). A well-behaved system allows the driver to stop the IST when the driver is unloaded. To achieve this, one of the data members of the structure passed to the IST is an abort flag, defined as volatile to prevent the compiler from optimizing/caching access to the variable because a different thread in the driver will alter its value to stop the thread. The abort flag is tested before and after waiting for the event, so that no further processing is done once the flag is set. (In a bit, we'll see how the thread gets out of the blocked Wait when we discuss stopping the IST.)

It is important for a well-behaved driver to consider the possibility of shared interrupts. Since the actual IRQ generating the interrupt is disabled until the IST calls InterruptDone( ), all other devices sharing the IRQ are blocked from getting any interrupts. Normally, the minimum requirement is to either clear or disable the interrupt at the physical device. This is all that is needed to prevent the system from re-detecting the same interrupt again. Once that is done, the IST can then re-enable the interrupts with a call to InterruptDone( ). It is also important to make sure that InterruptDone( ) is called, even when aborting the IST. Otherwise, the interrupt will remain disabled indefinitely, and all other devices sharing the IRQ will never get another interrupt.

DWORD DeviceIST(void *dat)
   ISTData* pData= (ISTData*)dat;
   CeSetThreadPriority(GetCurrentThread(), pData->Priority);

   // loop until told to stop
       // wait for the interrupt event... 
       WaitForSingleObject(pData->hEvent, INFINITE)
           // In case a real interrupt occurred

       // Handle the interrupt...
       // Use CEDDK functions to manipulate
       // hardware as needed...
       // doing as little as possible since the interrupt is still
       // disabled and other devices sharing the interrupt are blocked

       // Let OS know the interrupt disabled processing is done
       // this re-enables the interrupt so that other devices sharing
       // it can process their interrupts. 

       // Use CEDDK functions to manipulate
       // hardware as needed to complete the interrupt handling for the device
   return 0;

Typical IST Stop

A well-designed driver will provide the ability for the system to unload the driver—either from a plug-and-play removal or as part of a driver software upgrade. (Why force the user to reset the device just to update a driver?) In order for a driver to unload cleanly, you must design the IST to allow the driver to stop it before unloading. We've talked about this and have showed the creation and implementation of an IST that will support shutting down, so now let's look at what it takes to stop the IST.

The first step in shutting down the IST is to set the abort flag for the IST. This will pop the thread out of the while loop whenever it is not blocked in the WaitForSingleObject( ) call. This, of course, brings us to the next question: How do you get the IST out of the blocked call? That is one of the reasons for creating a data structure to pass to the IST. The handle to the event used in the wait call, and the logical ID associated with it, is all in the data structure. This allows the stop code to set the event to pop the IST out of the wait. However, to prevent any interrupts from occurring after the IST is shut down (or after releasing the event is no longer valid), the stop code must disconnect the event from the logical ID with a call to InterruptDisable( ). In most, but not all, cases this will cause the operating system to internally set the associated event to knock the IST out of its wait. Since there are a few rare conditions where that is not done, it is important to do it manually for safety.

Once the IST is triggered out of its wait state, it will see the abort flag has been set, perform any clean up, and return. The stop code can simply use WaitForSingleObject( ) with the handle to the thread to wait for the thread to finish. The operating system will set the signaled state of the thread handle when the thread exits. In a truly robust design, you would not use INFINITE as the timeout on the call to WaitForSingleObject. Instead, you would pick a reasonable value of a few seconds, and if that fails, call TerminateThread() to forcibly kill the thread or return an error code and refuse to unload.

When a process shuts down, all open handles and resources are cleaned up by the operating system. This is not the case when a driver unloads. This is because a driver is just a DLL loaded into one of the system processes (GWES.EXE or DEVICE.EXE), and the process itself isn't terminated. So, it is important to clean up any allocations a driver makes. This includes HANDLES, memory, and other resources. If you don't manually clean up a driver that is loaded and unloaded multiple times, this will cause a serious resource leak in the system that could, over time, lead to system failure.

// Set abort flag to true to let thread know
// that it should exit
g_DeviceISTData.abort = TRUE;

// Disconnect event from logical ID
// ( in most, but not all, cases this will
//   internally trigger g_DeviceISTData.hEvent
//   through the kernel)

// To be certain the IST gets out of wait
// manually set the event

// Wait for thread to exit
WaitForSingleObject(g_DeviceISTData.hThread, INFINITE);

// Clean up open handles

Shared Interrupts

Shared interrupts allow multiple devices to use a single IRQ, reducing the number of physical IRQs required for a bus. This requires alterations to the OAL and the driver to support the sharing of the interrupt. For each device that needs to use the interrupt, the OAL needs to have an ISR registered so that when an interrupt occurs, the OAL can go through the "chain" of interrupt handlers to find the actual source of the interrupt. (Or at least the first device generating an interrupt condition, if more then one is asserted at a time.) To support shared interrupts, the OAL must have a "static" ISR for the shared IRQ that calls NKCallIntChain( ) with the IRQ number. The kernel will run through each ISR registered for that shared IRQ. If the ISR returns a SYSINTR value other than SYSINTR_CHAIN, then that ID is returned from NKCallIntChain() so the Static ISR can return the logical ID of the interrupt.

Real-time Note   Due to the latency inherent in looping through an indeterminate number of ISR chains, you need to consider and account for the time required for that when building real-time systems. Unless you enable nested interrupts in the OAL, interrupts are off during this scanning of the ISRs.

If NKCallIntChain() gets through all of the ISRs and still gets SYSINTR_CHAIN as the return, then it's a spurious interrupt or a device generating an interrupt before it's ISR is added into the chain. This is a bad condition, and in many cases, depending on CPU and the interrupt controller, requires disabling the IRQ to prevent a permanent loop in the ISR. Since shared interrupts are level-triggered, they would perpetually trigger the ISR and nothing else would run. So it's important to make sure the device is not generating an interrupt in a shared scenario until the chained ISR is registered with the system. So, how does the chained ISR get registered in the system? There is only one documented method of doing this, and we'll see it in a moment.

The OAL normally implements all of the ISRs internally. (For some CPUs, like the ARM, there is really only one ISR.) However, with shared interrupts it is now possible for a device driver, or any application code, to load a DLL that implements an ISR after the OAL is built. For all interrupts that could be shared (or, if the OAL wants to allow a driver to install an ISR), as previously discussed there must be a small "static" ISR that calls NKCallIntChain().

The registration of an ISR in the chain occurs in a call to LoadIntChainHandler(). This function loads a DLL into the kernel's address space, and registers the specified exported function as the ISR for a specified IRQ. Due to the nature of how the DLL is loaded, it cannot have any implicit imports. (For example, it must not link to coredll.lib or any other import libraries, and it cannot use most of the C runtime support from coredll.) Microsoft provides a sample Generic Installable ISR (GIISR) that is useful in many cases (especially for PCI devices). There is also one for 16550 serial UARTS to create a software-based FIFO in the ISR to increase performance. You are, of course, free to create your own as needed by your hardware.

So there we have it, a walk through the interrupt architecture for Windows CE .NET 4.2, and a discussion of some of the more common questions and issues developers might have when thinking about implementing interrupt support on Windows CE .NET.


Get Embedded

Mike Hall is a Product Manager in the Microsoft Embedded and Appliance Platform Group (EAPG). Mike has been working with Windows CE since 1996—in developer support, Embedded System Engineering, and the Embedded product group. When not at the office, Mike can be found with his family, working on Skunk projects, or riding a Honda ST1100.

Steve Maillet is the Founder and Senior Consultant for Entelechy Consulting. Steve has provided training and has developed Windows CE solutions for clients since 1997, when CE was first introduced. Steve is a frequent contributor to the Microsoft Windows CE development newsgroups. When he's not at his computer burning up the keys, Steve can be found jumping out of airplanes at the nearest drop zone.