Export (0) Print
Expand All
Expand Minimize
2 out of 3 rated this helpful - Rate this topic

How Windows CE .NET is Designed for Quality of Service

Windows CE .NET
 

Microsoft Corporation

Updated February 2003

Applies to:
    Microsoft® Windows® CE .NET with Microsoft Platform Builder 4.0 and later

Summary: Learn about Windows CE .NET and Platform Builder design characteristics from the point of view of QOS and how it could relate to your development projects. In an effort to assess the quality of the Windows CE operating system and Platform Builder, Microsoft applies some of the industry-standard quality of service (QOS) guidelines used by other companies. (26 printed pages)

Contents

Introduction
Modularity
Scalability
Predictability
Adaptability
Stability
Security
Accessibility
Testability
Performance
Survivability
Conclusion
For More Information

Introduction

Microsoft® Windows® CE .NET is an operating system for small-footprint devices, typically diskless systems with limited memory capacity. Using a hard real-time kernel, the operating system provides a pre-emptive, multitasking environment with support for many technologies required by embedded systems. The customizable design supports a variety of resource-constrained embedded applications.

This paper assesses the quality of service of the operating system by evaluating a number of QOS characteristics. The following table lists the QOS characteristics discussed.

QOS characteristic Summary
Modularity Windows CE modularity is reflected in process isolation and memory protection that enhance system quality and performance, as well as efficiency of embedded system designs.
Scalability Scalability is evidenced by the fact that Windows CE is a true multitasking operating system that can accommodate many processes through a sophisticated thread-management mechanism.
Predictability Crucial to predictability is the rich set of hard, real-time capabilities offered by the operating system. Other aspects of the design that contribute to predictability are efficient handling of interrupt latencies, quantifiable system response and system timer access.
Adaptability The adaptability of Windows CE is supported by the modular design of the system. Adaptability is revealed through the Platform Builder features, including access to catalog component dependencies and shared source code.
Stability Windows CE shows its stability by operating robustly when stressed with large amounts of data and long periods of operation during continual long-haul testing.
Security Operating system security is supported by a number of services that ensure process validity, OEM certification and an enhanced redirector.
Accessibility Windows CE includes many design features that make its services accessible to processes.
Testability Windows CE is highly testable. The QA team for the product is dedicated to thorough testing of every product release. In addition, Platform Builder provides a complete set of testing tools that you can use to exercise embedded designs derived from the operating system.
Performance Windows CE consistently performs well in terms of speed and timing. To ensure this characteristic, performance testing is performed on an ongoing basis, for all releases of the product.
Survivability Windows CE shows its survivability in the face of software and hardware crashes, primarily in its power management design.

Table 1. QOS characteristics

Modularity

Windows CE and Platform Builder are based on a modular design that allows multiple programs to run at the same time as isolated processes. Each process executes in a fully protected memory space. The modularity of the operating system allows the core services to run in such a way that they do not interfere with applications, drivers and services. Modules and module components can be isolated and thoroughly exercised by rigorous testing.

Process Isolation

Each application executes through a single program instance, called a process. For each process, the operating system kernel keeps track of program counters, flags, registers and so on. The kernel uses these mechanisms to suspend the process as necessary. Windows CE supports a maximum of 32 simultaneous processes.

As shown in the next figure, the modular design of the operating system isolates processes so that they do not interfere with one another in cases of malfunction or crash. Each application has its own executable file, for example, MyApp.exe, which runs in the user process. Basic operating system support is provided by the kernel and the core system processes for the Graphics, Windowing, and Event Subsystem (GWES), the file system, drivers and the operating system services. Basic support is packaged in the image executable file Nk.exe.

Figure 1. Simplified Windows CE architecture

Process Memory

Windows CE is a virtual memory-based operating system that uses a memory management unit (MMU) to implement virtual memory and help protect processes. Since Windows CE systems usually have no disk drive, the operating system is designed to make maximum use of physical memory for long-term storage. Read-only memory (ROM), which is permanent, is used for the operating system image and any built-in applications. Random-access memory (RAM) is typically maintained continuously and can be used for persistent storage of the object store, which contains data or executable code not in ROM.

Use of virtual memory makes efficient use of memory and code space. Code can be executed in ROM, compressed in ROM, and paged into RAM, or stored in a file system and paged into RAM at execution time. Application code can also be placed in ROM, for example, for Microsoft WordPad. Additionally, built-in drivers reside in ROM.

The operating system supports one flat virtual address space, making available a total of 4 GB. Windows CE uses the lowest portion of the address space for process memory allocations, with one 32-MB slot dedicated to each process. The lowest slots, except for slot 0, are used for processes representing the modules; these include the GWES and file system modules that are part of the operating system executable file. These slots are reserved for system processes because they usually load first (although they do not have to do so). Slot 0 in the process space is always reserved for the currently running process.

Figure 2. Windows CE memory architecture

The next 1 GB of the address space is a shared memory area to hold objects accessed by two or more processes simultaneously. This area also holds large objects that cannot fit inside a particular process address space. The top 2 GB of the virtual address space are accessible only by the operating system for storage of its own data. For more about memory management in Windows CE, refer to Windows CE .NET Advanced Memory Management.

Thread stack memory

When a process creates a new thread, the kernel reserves RAM for the thread stack to hold function variables. The stack has a default, reserved size of 64 KB and built-in protection from stack overflow. The stack memory for a thread is reclaimed when thread execution terminates.

To make stack memory allocation more flexible, the kernel commits stack memory when the Scheduler handles a thread for the first time. Stack memory is committed one page at a time, only when needed. To prevent the initial commitment of stack memory from affecting system performance, a program should ensure that a thread is scheduled at least once before proceeding with real-time processing.

Heap memory

Windows CE also supports heap memory. Memory allocations in a heap are not made on a 64-KB boundary and are meant for allocations smaller than 96 KB. Use of the heap concept in memory management isolates processes from processor page size.

The kernel assigns a default heap for a process when it is created. The process can create separate heaps for different allocation uses to reduce the fragmentation that can occur when using a single heap.

Memory mapping

Windows CE supports memory mapping to enable multiple processes to share the same physical memory. Mapping dramatically increases system performance by allowing fast data transfers between processes. However, since memory-mapped files are designed for use by multiple processes, their memory is not protected.

The operating system uses a page-based mechanism to implement memory mapping. The use of paging ensures that background virtual memory management does not interfere with processing at real-time priority levels. Paging I/O occurs at a lower priority level than real-time priority levels, allowing paging in a real-time process to continue without interruption. However, real-time threads should be locked into memory as a safety measure.

Device Protection

Windows CE uses installable device drivers running in Device.exe to expose the services of peripheral devices, such as modems, printers, digital cameras and PC cards. Since the operating system uses installable drivers, device control depends on built-in hardware that must be connected to the Windows CE platform using serial ports, PC card slots or USB ports. Use of installable drivers ensures that driver code does not execute when devices are not being used.

Windows CE supports Services.exe, which is an alternative to Device.exe. If a driver crashes when running in Device.exe, it can corrupt other executing device drivers and cause the entire operating system to crash. However, you can design a driver to execute as a service to decrease the likelihood of an operating system failure in the event of a service crash.

Scalability

Scalability is the ability of a multitasking operating system to execute an increasing number of threads without degrading performance. Windows CE supports scalability by being a true multitasking operating system that uses a thread to represent each task. The operating system defines one or more independent threads of execution per process, with the number limited only by available memory.

Most user-mode applications are single-threaded, with all user and storage I/O handled by the single thread. However, more I/O-intensive applications frequently require multiple threads. For a multi-threaded process, one thread is designated as the primary thread.

Thread Synchronization

The Windows CE operating system uses a pre-emptive Scheduler that manages and synchronizes threads so that they can communicate and access the CPU as necessary. Each thread is assigned a context, consisting of an identifier, private stack memory and a set of registers. Threads of one process are not allowed to look at or alter the contexts of threads in other processes.

So that interaction among threads occurs correctly, the Scheduler must synchronize the threads using mechanisms called synchronization objects. These include critical section, event and mutex objects. Semaphores are fully supported. For example, you can use a named event as a binary control structure, or a semaphore for a maximum number control structure.

GWES manages message queues associated with threads, for either headless or display-based devices. Each thread can call GetMessage in the GWES API to create its queue, used in exchanging messages with a target thread.

When a process is created, Windows CE first allocates memory, including thread stacks, and creates the primary thread. Each thread of the process is given a default thread priority of 251 and a default thread quantum of 100 milliseconds. The thread quantum is used for round robin scheduling of threads at the same priority.

The Scheduler decides when to schedule a specified thread based on thread state and thread priority. Pre-emption is based solely on priority, and threads with higher priority are scheduled to run first. The Scheduler executes all threads of the same priority in a round-robin fashion, using the thread quantum set at thread creation. Lower-priority threads do not run until all higher-priority threads have finished, that is, until they either yield or are blocked.

Windows CE fixes thread priorities so that they do not change. The operating system does not age priorities and does not mask interrupts based on these levels. The Scheduler can temporarily modify thread priorities to avoid priority inversion. Priorities can also be changed at run time using the SetThreadPriority and CeSetThreadPriority functions.

Priority Inversion

Priority inversion is caused by a blocking, lower-priority thread holding a synchronization object required by a higher-priority thread. The thread Scheduler deals with priority inversion by using priority inheritance. Use of priority inheritance allows the lower-priority thread to temporarily inherit the priority of the higher-priority thread until it can run and free the resource for use by the waiting thread.

Earlier versions of the Scheduler had to search the entire thread pool looking for lower-priority threads that were blocking a higher-priority thread, and then temporarily boosting the priority of all blocking threads. This scheduling process was very time-consuming, tied up multiple threads, and led to unpredictable latency. The current version of the Scheduler solves these problems by limiting the temporary thread priority boost to one level below the blocked thread.

Thread Switching

The Scheduler can stop a thread from executing by switching it with another thread. Thread switching is necessary in cases of pre-emption by a higher-priority thread, expiration of the thread quantum, or the decision of a thread to block on a resource or sleep for a certain period. During a thread switch, the Scheduler stores the thread context of the stopped thread, starts the next thread and then restarts the waiting thread where execution ended.

Thread Blocking

Use of synchronization objects allows a thread to block its own execution and wait until a specified object changes. The operating system queues synchronization objects based on the priorities of the associated threads, with one queue for each priority level. The Scheduler adjusts these queues when priority inversions occur. See Priority Inversion.

Time Slicing

The Windows CE kernel supports a priority-based time slice algorithm for the Scheduler to use in thread execution. Time slicing is on a per-thread basis and is dependent on the default thread priority and thread quantum set at thread creation. You can modify the thread priority and thread quantum through calls to SetThreadPriority or CeSetThreadPriority and CeSetThreadQuantum.

Note   CeSetThreadPriority and CeSetThreadQuantum are protected API functions and can only be called by fully trusted applications.

Predictability

A predictable operating system is a hard real-time system for which processing is guaranteed to complete within specified time intervals. This section discusses Windows CE predictability in terms of these design aspects: real-time capabilities, interrupt latency handling and system response.

Hard Real-Time Capabilities

A real-time operating system, such as Microsoft Windows CE, must support asynchronous events that have time constraints generated by external events, environmental factors or machine interactions. For example, real-time systems are required by manufacturing process controls, high-speed data acquisition devices, telecommunications switching equipment, medical monitoring equipment, aircraft "fly-by-wire" controls, space navigation and guidance, laboratory experiment control, automobile engine control and robotics systems.

Any real-time operating system uses interrupts to ensure that it will quickly respond to real-world events. In terms of their handling of real-world requirements, operating systems can be categorized as follows:

  • Non-real-time operating system. Such a system is optimized for throughput, which is the total information to be handled over a definite period. It has an unpredictable interrupt latency between the time of an event and the time the operating system responds.
  • Soft real-time operating system. This system handles information and events promptly but has constraints based on the average time to handle events. However, it can tolerate occasional violation of these time constraints. An example of an application running on a soft real-time operating system is a graphical factory interface that needs to be updated promptly and regularly, but probably does not have any problem with a small amount of latency.
  • Hard real-time operating system. This system uses time as a hard constraint and measures a late response to an event as unacceptable. The interrupt latency between an external event and a system response must be reduced, and system response must be predictable and measurable. A hard real-time operating system must be interrupt-driven, support scheduling for multiple switchable threads, and allow access to the CPU for each thread. Mechanisms to support thread priorities and multitask communication are essential. A hard real-time operating system is frequently a multitasking system, but a system used for some fixed-function devices might perform only one task and still be hard real time. An example of such a system is an ABS brake-monitoring system.

The following table assesses the Windows CE design in terms of the most common hard real-time requirements. For more about the real-time aspects of the operating system design, refer to Designing and Optimizing Microsoft Windows CE .NET for Real-Time Performance.

Real-time requirement Windows CE design elements
Interrupt latency prediction and reduction The kernel design ensures predictable, bounded latencies for interrupts and associated threads. It also guarantees an upper boundary on the time required to start a real-time priority thread after receiving an interrupt. Profiling tools and utilities are available to record and analyze the maximum time required to process an interrupt. See Interrupt Latency Handling.
OEM control of interrupt handling The interrupt handling design permits an OEM to control interrupt processing with installable interrupt service routines (ISRs). The OEM has complete control over all interrupt service requests (IRQs) mapped to interrupt identifiers and associated interrupt handling software. See Interrupt Latency Handling.
Predictable system response The operating system response is predictable, with a system call time that can be validated. See System Response.
Thread access to system interval timer The thread Scheduler allows threads to access the operating system interval timer using the standard Win32 timer API functions. See Timer Access for Threads.
Thread context and synchronization The Scheduler assigns contexts to threads and uses predictable thread synchronization mechanisms. See ThreadSynchronization.
Thread priority support The Scheduler assigns thread priorities and supports priority inheritance. See ThreadSynchronization.
Thread switching The Scheduler can switch between thread contexts quickly and predictably. See Thread Switching.
Time slicing The kernel supports a priority-based time slice algorithm for the Scheduler to use in thread handling. See Time Slicing.

Table 2. Windows CE design common hard real-time requirements

Interrupt Latency Handling

The Windows CE kernel and OEM adaptation layer (OAL) are designed to optimize operating system response to external events within a specified time interval, using shared interrupts. Interrupt processing basically requires an interrupt service thread (IST) and an ISR.

The IST does most of the interrupt handling. It usually runs within a device driver and is suspended through a call to WaitForSingleObject. An ISR is used to direct the kernel to wake up the IST. Each hardware IRQ is associated with one ISR. If the CPU or additional associated hardware allows it, the operating system nests ISRs based on priority to prevent the loss and delay of high-priority interrupts.

Interrupt latency is the amount of time between the arrival of an external interrupt at the CPU, by way of IRQ, and the time the IST starts. The latency actually consists of two separate sub-latencies: the time between the external interrupt and the start of the ISR (ISR latency), and the time between the start of the ISR and the start of the IST (IST latency). To allow for quantifying and reducing interrupt latencies, Windows CE bounds interrupt latency times for a thread with its stack and all of its memory locked against paging. Bounded latency times allow a program to calculate worst-case latencies, the total times to the start of the ISR and to the start of the IST. The program can then determine the total amount of time until an interrupt is handled by calculating the amount of time needed within the ISR and IST. Windows CE includes tools for measuring interrupt latency on an embedded system so that you can verify that thread priorities are leading to well-defined behavior.

System Response

In Windows CE, the operating time is predictable for every part of the kernel that cannot be pre-empted. If a kernel component runs for only a bounded time, thread switch latencies are bounded. The time required is independent of the number of processes, threads and synchronization objects in the operating system. You can validate the time using a special instrumented build of the kernel.

Timer Access for Threads

Windows CE uses a system interval, or system tick, timer to define the rate at which the system generates and services an interrupt. The timer has an accuracy of 1 millisecond in timer and Sleep function calls. The thread Scheduler permits thread access to the system interval timer using the standard Win32 timer API functions, which use software interrupts from the kernel to obtain time intervals to use in managing real-time applications. For example, a call from a thread to GetTickCount obtains a count in milliseconds from the interval timer.

If desired, you can implement a higher-resolution timer using the applicable OAL interfaces. The OEMIdle function, which is called by the thread Scheduler when there are no threads to schedule, can be used in this implementation. This function typically reprograms the interval timer based on the amount of idle time, and puts the CPU in standby state to preserve power.

Adaptability

An adaptable operating system is one that can be extended and enhanced. Windows CE is a highly adaptable operating system due to its modular design, introduced in Modularity. The adaptability of the operating system is exposed through Platform Builder, which provides ideal support for embedded system development.

Support for Catalog Component Dependencies

The Windows CE Catalog represents all operating system features, board support packages (BSPs), platform configurations, transport layers and drivers in the form of components that are based on the modular design of the operating system. You can also add third-party components and make them available through the Catalog. Sample drivers, services and BSPs are provided for you to modify as needed.

Platform Builder organizes and displays all components in a selectable format that allows you to choose only the components needed as the basis for an embedded design. Thus, you can limit the applications and drivers required for a specific design, which reduces the overall operating system footprint.

Shared Source Code

Microsoft allows access to almost all of the shared source code for Windows CE. This code is furnished under the Microsoft Shared Source License for Windows CE .NET and ships with every version of the operating system, including the Emulation Edition. Code modules include: Explorer Shell, Web server, Simple Object Access Protocol (SOAP) implementation, Universal Plug and Play (UPnP) protocol implementation, Microsoft Message Queue (MSMQ) Server for Windows CE, wireless network drivers, expanded file system and storage code, and expanded kernel code. The Shared Source for the Windows CE .NET Test Kit is also available as a download.

Use of shared source code can aid in the development of Windows CE-based designs in the following ways:

  • The source code exposes the inner design of existing operating system components. Exposure of the design simplifies the development and debugging of applications, drivers, and services that work with the operating system.
  • Comparing the derived source code with original source code can aid in diagnosing problems with a platform in development.
  • Studying the source code configuration for an existing hardware component helps in configuration of a new hardware component.
  • Access to source code allows you to provide useful feedback to Microsoft about developing platform technologies and designs.
  • You can modify the source code and redistribute it in noncommercial scenarios, such as academic and research projects and individual development efforts. This redistribution encourages collaboration and innovation among developers.
  • You can modify shared source code and use it for reference and debugging in commercial situations.

Stability

Stability can be defined as the ability of an operating system to provide consistent operation over a long time, challenged by heavy data loading. Stability has become an important part of acceptance criteria for final signoff and release of Windows CE and Platform Builder. To verify how well the operating system and Platform Builder stand up to operational stresses, Microsoft routinely tests the products using simple scenarios that are expected to run for long periods and tolerate large amounts of data without crashing. For details of Windows CE QA team methods used in long-term testing, see Long-Haul Testing.

Security

A secure operating system provides protection against malicious and inadvertent unauthorized access. Protections must be provided both for core services and for external applications and dynamic-link libraries (DLLs). The Windows CE design includes a variety of security provisions. This is critical because the operating system supports many mobile information appliances, with a variety of programming interfaces that are at risk from data tampering, unauthorized user, and rogue applications.

Security in Windows CE is based on a trust model that helps protects against internal attacks from a code module running on the local device, for example, by preventing one process from reading or modifying the memory of another. The trust model supports the following privileges for each code module:

  • Full trust. The module has full privileges for operating system use.
  • Partial trust. The module has partial privileges. It cannot modify portions of the registry or process threads, set kernel mode, or perform other protected operations.
  • Not trusted. The module is not trusted and is not placed in memory to execute.

A system that does not implement the Windows CE trust model is not protected against internal attacks. However, the operating system kernel does help protect against external attacks from remote sources, such as attempts to read data on the device from a remote computer. The operating system also implements some protection against using a device to attack remote computers, for example, other computers on the same network.

Security Services

Windows CE provides services to help protect communication mechanisms, user applications, file systems and data store, and network management for all supported interfaces. The following table shows the services that are used to verify the validity of processes and their permissions to access the operating system.

Security service Windows CE security mechanism
Authentication Password Authentication Protocol (PAP), Challenge Authentication Protocol (CHAP), and Microsoft CHAP for authentication at the network datalink level; Kerberos and Windows NT LAN Manager (NTLM) authentication protocols for enhancing the security of network connections
Data store protection Protected store API for cryptography and key management
Encryption/decryption CryptoAPI cryptography services
Integrated security services Security Support Provider Interface (SSPI) to request an authenticated connection from a security provider; supported providers are NTLM and Kerberos
Operating system environmental protection Trusted environment model, in which only certified applications or DLLs can run
Public key infrastructure X.509 certificate management and verification services through CryptoAPI
Secure sockets Secure Socket Layer (SSL), through Windows Internet API (WinInet) or Winsock
Smart card support Windows CE smart card subsystem, supporting CryptoAPI and Windows CE-based device driver model

Table 3. Services used to verify validity of processes and permissions to access the operating system

OEM Module Certification

Windows CE builds capabilities into the OAL to certify processes and DLLs for access to operating system services and privileges to execute. The operating system loader must verify privileges for any module not in the basic operating system ROM image.

The OAL uses the OEMCertifyModule function for process and DLL certification. This function enables the operating system loader to communicate with initialization software to ensure certification. You can customize the loader to certify applications and DLLs as required.

Use of the Redirector in Security

The Windows CE networking design uses the redirector, which is a network file system driver providing access to files on remote computers. To minimize security risks, the redirector helps protect shares with credentials supplied by the user or an application through the WNetAddConnection3 function. The redirector uses these credentials for authentication when it first connects to a server. While the server session remains connected, other applications can access files with the same level of access as the original logon. Once all open files have been closed, however, the redirector shuts down the session, making it necessary to authenticate subsequent accesses again. If an application supports user switching, it can also use the WNetEnumResource and WNetCancelConnection2 functions to force the closing of all open files on a specific server and help protect connections established by earlier users.

Accessibility

Windows CE uses a number of design mechanisms to make its services accessible to processes that need them. As an illustration, this section describes the network access services provided by the operating system.

Windows CE is accessible by a variety of networked devices. Wireless support is furnished for personal area networks (PANs), including Bluetooth and IrDA protocols; local area networks (LANs); and wide area networks (WANs), including Bluetooth and 802.11 protocols. Wired support is included for PANs. To accommodate network-enabled applications, the operating system supports Internet Protocol version 4 (IPv4) and Internet Protocol version 6 (IPv6).

If a network accesses the Internet, it will use the Windows CE-based Web server to fulfill requests for information from applications, for example, a browser. This information will be delivered using the HTTP protocol. To help protect the network when the Web server is being used, Windows CE runs the server in Services.exe, instead of Device.exe. Isolating the server execution allows the server to crash without disturbing the operation of other networked devices.

Testability

Windows CE is thoroughly testable, with testability enhanced by the modular operating system design. The Windows CE QA team at Microsoft subjects the operating system and Platform Builder to rigorous testing during the preparation of each software release. In addition, you can use Platform Builder-supplied test tools to examine Windows CE-based designs.

Microsoft Windows CE Testing

The Windows CE QA team at Microsoft runs a variety of tests on each operating system version to ensure the highest quality possible. Listed below are the main factors that the QA team considers in planning and executing operating system tests:

  • Large number of components. Windows CE includes over 200 components. Therefore, the QA team cannot confine itself to just one kind of operating system testing, but must use several test methodologies to get separate measures of quality. The team uses these measures to corroborate one another, leading to more confidence of product quality.
  • Many component combinations. Windows CE allows embedded designs that put operating system components together in many combinations suitable to a variety of markets and applications. Thus, a great number of component combinations require testing. Dependencies among components make most of the combinations non-operational, but a large number of valid combinations remain to be examined.
  • Example operating system configurations. Microsoft knows that you are not interested in picking each individual operating system component from scratch when building an operating system image. Therefore, Windows CE ships with a number of example operating system configurations. The QA team focuses on these configurations for testing.
  • Globalization support. The Windows CE QA team must consider globalization in its testing of Windows CE. This type of testing adds a measurable amount of time for the QA team, especially for Asian languages that add an input method editor (IME).
  • Several processors. Windows CE currently runs on processors in a number of major microprocessor architecture families.
  • Large amounts of test code. Testing must covers many different operating system builds, with several million lines of test code. The QA team facilitates this testing by taking into account code that is the same across the different processors and checking the few processor-specific bugs. Not all tests have to be run on all configurations.

The following illustration shows the Windows CE QA team test cycle used for each new operating system feature. Altogether, the QA team spends about twice as many hours ensuring quality as it does creating features. This estimate does not include the hours that the Microsoft Tools team and strategic partners put into testing the operating system before it is examined by the QA team. Neither does it reflect the testing done by the various product teams who build on top of Windows CE.

Click here for larger image

Figure 3. Microsoft test cycle for Windows CE (click thumbnail for larger image)

Windows CE QA testing is a significant portion of the project timeline and human energy invested in the product. It requires a large number of testers and thousands of test cases for each product version. To run all the tests takes the QA team about four weeks, assuming no delays.

Internal test tools

The following table lists the test programs that the Windows CE QA team uses for internal product testing. Along with these tools, the QA team also uses the Platform Builder tools summarized in Embedded Design Testing.

Test tool Description
Microsoft Windows CE .NET 4.2 Test Kit (CETK) Tests peripherals and drivers under development with minimal Microsoft involvement. Several tools are provided, with access to the shared source code. For more about the CETK, see Building and Testing Devices with the Windows CE .NET Test Kit (CETK).
CEStress Thoroughly exercises a broad set of platform components for an extended time. The modular tests run by CEStress cover all stress testing for the product.
Coverage tools Scan code and point out possible errors before the code is executed. Microsoft developers run the tools and fix any problems before submitting code in the source tree. In addition, the build team uses the tools daily to check all of the operating system code.
Kernel Profiler Determines the functions that are taking up the most operating time on a device over a specified period. This tool is a statistical profiler that has some of the functionality of the Platform Builder Remote Call Profiler tool, but does not require an instrumented build of the modules. It also does not affect system behavior as much as Remote Call Profiler does. The Kernel Profiler tool runs only on a Windows CE-based target device.
OEMStress Exercises platform components for an extended time. This tool provides a subset of the CEStress functionality. It is included in the CETK.
StatCollect Collects statistical data during testing and stores the data in a file. This tool also allows for reporting results to the currently running Windows CE long-term server (CELT). Data collected by StatCollect includes active TCP connections, ports on which a computer is listening, Ethernet statistics, IP routing table, IPv4 statistics and IPv6 statistics.

Table 4. Windows CE QA team test programs

Application Testing

The Windows CE QA team tests all existing and new applications shipped with each Windows CE release. Examples of these applications are Microsoft Internet Explorer and ActiveSync. The basic purpose of application testing is to ensure that new kernel technologies and performance enhancements do not adversely affect existing applications. Testing also includes checks for backward compatibility.

Automated Testing

The Windows CE QA team must carefully test all Windows CE configurations as described in Configuration Testing. To keep configuration testing time to a minimum, the team uses automated test programs that combine individual tests for a configuration and execute with minimal human assistance. The goal for each configuration is to have an automated test program that detects the configuration components, at compile time or run time, and adjusts itself automatically to compensate.

Beta Testing

The Windows CE QA team does not usually do formal beta testing during the development of Windows CE. However, for virtually every version of the operating system, the team does furnish "as-is" (unsupported) software previews for partners who cannot wait for release to market. The QA team uses these previews to obtain valuable customer feedback and follow up on the promise of better communication with customers.

Microsoft has enhanced informal beta testing to include newsgroup participation, which allows the QA team to scan for customer problems that can translate to operating system bugs. During beta testing, customers are also presented with questionnaires targeting specific operating system areas in an effort to find features to improve.

Browser Testing

Windows CE supports Internet services for these browsers:

  • Microsoft Internet Explorer 6.0 for Windows CE
  • Microsoft Pocket Internet Explorer

The operating system also supports WinInet, which allows the addition of Internet client functionality to applications.

The Windows CE QA team tests all aspects of the Windows CE browsers, including: ActiveX objects, anchors and so on. The following are some specific design areas that must be tested:

  • HTML tags and attributes
  • Cascading style sheet (CSS) keywords and attributes
  • Dynamic Object Model (DOM) for HTML, which contains a number of methods and properties for each object
  • Interactions among the above pieces on one page

Each file is specifically tested and considered a test case. The number of tests is in excess of 148,000.

The Windows CE QA team tests the browser in progressively strict and comprehensive phases. The general phases are smoke-level testing with light coverage to verify basic functionality, functional testing and stress testing. The team identifies areas within each phase and tests them formally. After formal testing is complete, the team performs ad hoc tests. Full test passes are made for U.S. English, German, and Japanese languages. Smoke-level tests are performed for all other languages supported by the operating system.

Testers create test pages exercising specific browser functions in a formal, measurable way. Additionally, they test a dynamic number of popular sites, including public Microsoft sites, competitors' sites, and remote and foreign sites. The total number of popular sites visited in a typical product cycle of less than 6 months is around 1000.

Note   Since Pocket Internet Explorer does not have all the features included in other Internet Explorer browsers, its testing applies only a subset of the methods described above. For example, testing for Pocket Internet Explorer does not need to exercise CSS functionality.

Configuration Testing

The Windows CE QA team is responsible for testing all Windows CE platform configurations. These configurations must be tested for all processors and for all languages.

There are many individual, and time-consuming, tests for each available configuration. To reduce configuration testing time, the QA team frequently uses automated test programs, as described in Automated Testing. Not all configuration tests can be automated. However, the QA team makes every effort to automate as many of these tests as possible, using the time saved to increase the number of automation servers and direct queries and improve information logging.

Functional Testing

The Windows CE QA team must test each Catalog feature shipped with Platform Builder, at the function level. Many operating system API functions are designed to be independent of the current state of execution. For example, an API function should behave identically any time it executes. Functional testing verifies consistency by allowing random permutations in several test suites. Results are checked independently from the current state of the program.

The QA team uses functional testing to demonstrate that each feature component operates as it should and has no harmful side effects. For each release, testers must prepare a large number of test suites. Some tests are manual operations conducted by a human tester, but the majority are automated software test routines written by the QA staff. The following table summarizes the types of tests used in functional testing.

Functional test Description
API test Uses targeted test cases to verify conformance to published API specifications for all configurations and platforms. Any differences must be clearly documented and delivered in the SDK. API implementation details and platform differences should be transparent.
Boundary analysis test Scans through the most probable error scenarios by targeting value boundaries, instead of the values themselves. For example, integer boundaries are negative numbers, MAXINT, zero and so on. Boundary analysis tests are useful when the number of possible API parameter values is large.
Build verification test (BVT) Examines the basic operation of an API function, addressing the most common parameters, boundary parameters, and invalid parameters. The QA team uses a BVT to find obvious or fundamental errors and to verify that a component is ready for additional testing. If major functional problems are detected, they are fixed before additional testing is performed.
Calculation error test Verifies calculations by API functions that do simple arithmetic. Testing uses boundary analysis and checks for valid and invalid parameters.
Error handling test Runs each API function through a series of bad, null, and invalid parameter tests to ensure overall operating system robustness. The error-handling test verifies the compatibility of error codes and extended errors with the Win32 specification.
Initial and later state error test Ensures correctness after repeated execution of the same API function.
Regression test Uses a combination of functional testing approaches to target a specific bug fix. When a bug is discovered and fixed, it is placed into circulation within the regression test suite. A regression test is run repeatedly through the development and validation portions of a project, and again during the release cycle, to ensure that the operating system remains stable and is ready to ship. To make the test more comprehensive, additional test cases are added for bugs found by other means.
Smoke-level test Verifies basic functionality. This test is slightly more comprehensive than a BVT.

Table 5. Tests used in functional testing

Globalization Testing

Globalization testing is essential for every Windows CE Catalog feature, and is explicitly spelled out in each feature test plan. The Windows CE QA team uses the globalization tests defined in the next table. The test suite consists of about 900 tests of the display, the control panel, networking, storage and so on. The same tests can be performed for any language, with the exception of the IME.

Globalization test Description
Global functionality test Verifies support for international functionality in the core product. This test examines multi-byte characters, boundary characters, conversions from Unicode to a double-byte character set, and IME testing. Testers perform a feature-level test of German and spot checks of other languages.
International sufficiency test Verifies that the international features built into the U.S. English product are correctly supported.
Localization test Ensures that translations and associated user interface layout changes are correct and that a translated version of Windows CE operates correctly. The Windows CE localization test team checks each language and user interface.

Table 6. Globalization tests

Long-Haul Testing

Long-haul testing is used by the Windows CE QA team to exercise a set of user scenarios that are allowed to run until crashing. This type of testing is ongoing and routinely uncovers many bugs that might be missed by the functional, performance and stress testing that is the normal part of any software release.

The QA team sets up long-haul testing scenarios that determine statistically how a device is most commonly used. The scenarios project enough resources to satisfy requests, repeated over a long period. Accommodation of an individual request might take seconds or minutes for an average device. However, long-haul testing determines how the device handles the request repeatedly over days and weeks. Testing must ensure that the device remains operable throughout the test period.

Long-haul test scenarios reflect the platform configurations accessible through Platform Builder. The QA team can also design custom scenarios to include variations for individual user setups.

An important use of long-haul testing is to locate resource leaks that are not easily found during development testing. For example, even though a 4-byte handle leak is negligible during one test cycle and usually does not affect the operation of a specified task, it can represent a significant memory drain when accumulated over thousands of test repetitions. The QA team does not consider any memory leak acceptable.

Performance Testing

Performance testing ensures that changes made to Windows CE and Platform Builder do not adversely affect system performance. The Windows CE QA team must verify that a modified operating system performs at least as well as it has in previous releases. Performance numbers used as standards in the testing are hardware-specific, depending on CPU type and speed, memory architecture, and cache organization and size.

For performance testing, the QA team uses a group of testers dedicated specifically to developing and executing performance tests. Use of a dedicated testing group allows test execution in a controlled and standardized environment for each software release.

Performance tests examine all major areas of the operating system, including the kernel, GWES, drivers, networking stack and file system on all processors. Testing is done at a high level to ensure that the operating system as a whole is tested, instead of just individual features. The testers must evaluate performance under various stress levels, which requires running from one to hundreds of threads of varying priorities on the test device.

Performance testers use a variety of user scenarios that contribute to overall operating system results. During testing, they log bugs against significant performance slowdowns and track progress until the bugs are resolved. The performance test cycle continues as long as new builds are introduced.

All performance test results are logged in an SQL database that supports queries and reports to show progress over time. The performance testers make weekly reports to the rest of the QA team, taking into account changes from baseline builds, as well as changes from results gathered the previous week. The weekly reports help testers better understand operating system revisions and track down anomalies.

For more information about performance test methodologies, see Performance Test Methodologies for Windows CE .NET. This document describes user scenario tests, as well as introducing low-level performance tests that take explicit steps to examine the performance of individual operations not exercised in user scenario tests. These low-level tests are particularly helpful in situations that place a particular subsystem under high stress. Low-level performance tests include the use of such industry-standard equipment as Smartbits by Spirent and VQT by Agilent. The QA team tracks low-level performance of operating system components with such standard tools and other specialized custom-built tools to ensure the high quality of the operating system.

Plugfest Testing

Plugfest testing analyzes the interoperability of operating system features and devices in every major industry initiative. The Windows CE QA team uses plugfest testing to put early versions of feature code through interoperability tests against OEM hardware and industry requirements. For this type of testing, the team uses either Windows CE-specific plugfests organized and sponsored by Microsoft or plugfests supported by industry groups.

Real-Time Testing

Real-time testing for Windows CE measures and tracks crucial interrupt latencies. This type of testing also examines the time durations of system functions linked to real-time performance, for example, signaling a semaphore or waiting on a system object.

Real-time response of the operating system is greatly affected by your setup of the hardware platform and the implementation of the OAL that you provide. To assist you in real-time testing that is compatible with internal Microsoft testing, source code for the ILTiming and OSBench test tools is included with Platform Builder. See Embedded Design Testing.

Security Testing

Security testing for Windows CE is ongoing at Microsoft and includes tests of the trusted security model, the Protected Library Server (PSL) stack, password storage, data protection APIs, and so on. The Windows CE QA team regularly scans operating system code and new feature source check-ins and checks them for buffer overflows. The testers also examine the source code against a list of security vulnerability categories.

The QA team gives careful attention to servers and other connectivity mechanisms, because they can allow remote attacks on the operating system. These mechanisms are regularly tested against common hacker attacks known at this time.

Size Testing

Size testing is used by the QA team for each Windows CE version to determine the effects on module size of new features, componentization, and dead code. Currently the team gathers the sizes of all .dll and .exe files from daily builds and reports on their growth measured against previous builds. Testers track size increases and expand targeted development efforts to reduce size as required.

Stress Testing

The Windows CE QA team uses stress testing to ensure that Windows CE continues to operate under all types of system activity and with many applications active at the same time. Run on a number of reference platforms for each supported CPU, stress tests have to examine underlying hardware as well as the operating system. QA team states the results of stress tests in terms of mean time to failure (MTTF). With each release of the operating system, the QA team has increased the MTTF number that must be met for shipment.

For stress testing, the QA team uses the CEStress tool, which examines an arbitrary Windows CE platform in an environment that intensely exercises a broad set of system components for an extended period. This test is responsible for the following:

  • Verifying the stability of individual component functions after prolonged use. For example, the test determines if an individual component leaks memory, crashes, stops responding or functionally fails after continuous operation.
  • Verifying the health of components after prolonged use.
  • Verifying the interaction of system components in arbitrary combinations. For example, the test determines if components deadlock, time out, or functionally fail when competing for system resources.

CEStress consists of a number of individual test modules. Each module is targeted at a specific functional area of the operating system, for example, the file system or Dialog Manager. A module is capable of running a number of tests on the targeted component, from simple API calls to small user scenarios. The module tests are run in random sequence in a loop for an arbitrary amount of time.

Stress testing uses a test harness, which is a single executable file that runs on the device. It is responsible for launching the modules, managing their lifespan, and recording their results. In addition, the harness monitors various system statistics, including memory use, and reports the state of the operating system and the cumulative results to the tester.

The nature and intensity of the stress-testing environment depend on the following parameters of the test harness:

  • Module mix. By default, CEStress runs the largest subset of modules that correspond to the platform's software configuration. However, the combination can be modified to focus on individual components, or to add modules.
  • Number of concurrent modules. The test harness attempts to keep a specified number of modules running concurrently at all times. As modules finish and exit, they are replaced by new module instances, selected at random from the module mix. By running a number of concurrent modules, the QA team can intensify the focus on interoperability testing.
  • Module duration. The test harness can make modules run for any amount of time. Short durations allow a focus on repeated create or destroy, connect or disconnect, and initialize or cleanup scenarios. Long durations allow testing for the accumulation of resources. By default, the test harness sets a module duration of 10 minutes, which allows a reasonable combination of both short-duration and long-duration stress testing.
  • Test duration. Because stress testing constantly generates unique test combinations, the longer CEStress runs, the more likely it is to confirm that the platform is robust, or to encounter potential bugs. The QA team uses a default test duration of 900 minutes (approximately overnight) to generate sufficient problems for an adequate test.

Embedded Design Testing

Platform Builder comes with a variety of test tools that you can use to verify quality in your Windows CE-based designs. The following table lists the main test tools in Platform Builder.

Basic test tool Description
Application debugger Debugs applications on a customized operating system image.
CETK Tests peripherals and drivers under development with minimal Microsoft involvement. See Internal Test Tools.
ILTiming Analyzes interrupt timing and measures interrupt latencies.
Kernel debugger Debugs a customized operating system image and provides information regarding the performance of the image.
LMemDebug Accesses data about operating system memory allocations. Memory debugging hooks can be used for memory leak tracing and performance analysis. This tool is available for both headless and display-based devices.
OSBench Supports the collection of timing samples to measure kernel performance. This tool is used for scheduler timing analysis, focusing on the time required for basic kernel operations. It supports thread Scheduler testing in the same process or between separate processes.

Table 7. Platform Builder main test tools

The following table lists the remote test tools in Platform Builder. These tools allow you to use a remote development workstation to perform a variety of programming tasks on a Windows CE–based target device.

Remote test tool Description
Remote Call Profiler Examines and maps software interactions in an operating system. This tool enables profiling and analysis by identifying algorithmic bottlenecks in applications. It is supported only for platforms created with Platform Builder 4.0 or later.
Remote File Viewer Displays a hierarchical view of the file system on a target device.
Remote Heap Walker Displays information about heap memory identifiers and flags for processes running on a target device.
Remote Kernel Tracker Represents the execution of the operating system on a target device. In a single view, the tool presents the activities of all threads on the target. It allows viewing of thread interactions, internal dependencies, and system state information in a real-time environment.
Remote Performance Monitor Measures the performance of a Windows CE-based platform. This tool enables tracking of current activity on a target device and makes data accessible in a log file. The tool permits observation of the behavior of objects such as CPUs, threads, processes, system memory, and associated performance counters. Counters provide information about device use, queue lengths, and delays, and information used to measure throughput and internal congestion.
Remote Process Viewer Displays information about the processes running on a target device.
Remote Registry Editor Displays the registry for a target device and enables registry management.
Remote Spy Displays a list of windows opened on a target device. In a separate window, the tool shows information about the messages in the message queue for the selected window.
Remote System Information Displays system settings and properties for a target device.
Remote Zoomin Displays an image from the screen of a target device.

Table 8. Platform Builder remote test tools

Performance

Operating system performance is largely defined by the speed and efficiency of system operations. CPU speed, memory bus speed, and graphics hardware are a few of the variables that have a major impact on performance.

Windows CE is carefully designed for performance. Because hardware varies from device to device, Windows CE includes a number of variables that can be adjusted to ensure the best possible performance of an embedded system.

To ensure performance, each Windows CE release must be tested rigorously to prevent performance degradation from added technologies, hardware or tools. Software for a new release must perform as well as in earlier versions, if not better. Windows CE performance testing is described further in Performance Testing.

Survivability

Essential to fault tolerance in an operating system is its ability to withstand hardware and software crashes. This section discusses the Windows CE power management design to illustrate survivability of the operating system.

Power Management Design

Windows CE provides several mechanisms for power management and handling of power-related faults. The core operating system uses several power states to handle assorted power management tasks. Power states include: no power from any source, on (full power), critical (battery power), system idle (no power), suspend (enough power to sustain RAM), and reset.

In addition to core operating system power management, Windows CE supports power management through its OAL, and allows GWES to manage power for user inputs. An optional Power Manager can be used to enhance basic power management by the core operating system.

The main goals of power management in Windows CE are:

  • To use batteries wisely, so that battery-operated devices can provide service for long periods of time
  • To help protect the object store, which resides in volatile RAM, from data loss
  • To handle suspend conditions gracefully
  • To allow enhancement of basic power management capabilities

Power Protection for Battery-Powered Platforms

Windows CE uses the system idle power state to allow a battery-powered platform to turn off when no user activity has been detected for a specified time. GWES monitors touch screen, mouse, and keyboard activity. If no user activity occurs during a specified time, the device can switch to idle. You can control the time through applications that call SystemIdleTimerReset.

Object Store Protection

The operating system helps protect the object store against unauthorized access or modification by protecting the RAM used to hold the object store. However, because the object store is stored in RAM, its data is erased if there is no power to the device. For that reason, you might choose to use a backup battery or to store data outside the object store in nonvolatile storage, such as flash memory or a hard disk.

Handling for Suspend Conditions

To manage power for user inputs, GWES sets a timer to watch for user activity and suspends the operating system when it sees no activity for a specified time. When in the suspend power state, Windows CE can be awakened under application control, by user action, or under hardware control.

An application can handle suspend conditions by using registry entries to set timeout values for battery power and system connection to external power, among other values. The application can also disable GWES power management with a registry key entry.

Hardware control of suspend conditions is also possible in Windows CE. Suspended devices frequently need to operate in an "instant-on" state when they resume operation after a suspend period. The operating system supports instant-on operation by keeping all programs frozen in memory during a suspend power state. This allows the system to turn on quickly when the user hits the power button or another wakeup event occurs.

The OAL power management design allows an installed device driver to be set up to receive a notification when the operating system enters the suspend power state. Because this notification occurs in an interrupt context, devices are severely restricted during the suspend state as to what they can do and how long they can take to do it. The OAL allows an interrupt associated with an installed device driver to be defined as a system wakeup. An installed interrupt handler can also be set up as a wake source.

Enhanced Power Management

In the basic Windows CE power management model, applications get no warning when the power state is changing to suspend. Some device drivers are given a notification, but they are tightly constrained in their operation during the suspend period.

Support for the optional Power Manager solves this problem by allowing applications to receive notifications of changes in the status of system power. To implement this operating system component, you must include the PM.dll file in the platform build.

Power Manager allows devices to manage their own power as long as they keep their power levels between the maximum and minimum limits. If the minimum power consumption limit for a device is set higher than the maximum, the power of the device remains elevated for as long as the application requires the device.

Conclusion

This paper has introduced the subject of software QOS, as it applies to Windows CE and Platform Builder designs. It has assessed the operating system and Platform Builder according to a variety of QOS characteristics to help you apply similar concepts in your Windows CE-based development projects.

For More Information

For the latest MSDN® information about Windows CE, see Microsoft Windows CE .NET.

The online documentation and context-sensitive Help included with Windows CE also provide comprehensive background information and instructions for using the operating system.

To access the online documentation for Windows CE .NET

  1. Start Platform Builder.
  2. Select the Contents tab on the Help menu to view the documentation.
Did you find this helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft. All rights reserved.