Operating Systems and PAE Support
Updated: July 14, 2006
On This Page
PAE: 32- vs. 64-Bit Systems
Addressing physical memory above 4 GB requires more than the 32 bits of address offered by the standard operating mode of Intel (32-bit) processors. To this end, Intel introduced the 36-bit physical addressing mode called PAE, starting with the Intel Pentium Pro processor.
This article describes some techniques that Microsoft Windows operating systems and several UNIX operating systems use to provide support to applications using PAE mode addressing. Because processes running in these environments have 32-bit pointers, the operating system must manage and present PAE's 36 bits of address in such a way that the applications can practically use it. The key question is: how does the operating system solve this problem? The performance, functionality, simplicity of programming, and reliability of how these issues are handled will determine the usefulness of the large memory support.
PAE is supported only on 32-bit versions of the Windows operating system; 64-bit versions of Windows do not support PAE. For information about device driver and system requirements for 64-bit versions of Windows, see 64-bit System Design. The Address Windowing Extension (AWE) API is supported on 32-bit systems. It is also supported on x64 systems for both native and Wow64 applications.
Although support for PAE memory is typically associated with support for more than 4 GB of RAM, PAE can be enabled on Windows XP SP2, Windows Server 2003, and later 32-bit versions of Windows to support hardware-enforced Data Execution Prevention (DEP).
The information in this article applies to Windows 2000, Windows XP Professional, Windows Server 2003, and later versions of these operating systems, referred to as "Windows" in this paper.
Address Translation in standard 32-bit mode
Windows uses two levels of mapping to do the translation, which is facilitated by a set of data structures called page directories and page tables that the memory manager creates and maintains.
PSE uses a standard 1K directory and no page tables to extend the page size 4-MB (eliminating one level of indirection for that mode). The Page Directory Entries (PDE) contains 14 bits of address, and when combined with the 22-bit byte index, yields the 36 bits of extended physical address. Both 4-KB and 4-MB pages are simultaneously supported below 4 GB, with the 4-KB pages supported in the standard way.
Note that pages located above 4 GB must use PSE mode (with 4-MB page sizes).
Windows supports PAE with 4-KB pages. PAE also supports a mode where 2-MB pages are supported. Many of the UNIX operating systems rely on the 2 MB-page mode. The address translation is done without the use of page tables (the PDE supplies the page frame address directly).
Operating System Implementation and Application Support
The next issue is how the operating system can manage and present PAE's 36 bits of address in such a way that an application (with 32-bit pointers) can practically use the additional memory.
There are five application support models. The first two models (Server Consolidation and Large Cache) are completely handled within the operating system and require no changes to the application. The second two models (Application Windowing and Process Fork) require application changes to support API extensions for large memory. The last model (PSE 36 RAM Disk) requires no changes to the operating system (implemented in a driver), but mandates application changes to support the driver.
1. Server Consolidation
With the additional physical memory supported under PAE mode, an operating system can keep more of these applications in memory without paging. This is valuable in supporting server consolidation configurations, where support of multiple applications in a single server is typically required. Note that no application changes are required to support this capability.
2. Large Cache
3. Application Windowing
Sharing high memory between processes can introduce quite a bit of complexity into the API and the implementation. Windows avoids this kind of sharing.
In addition, the support of paging makes the design and implementation of the operating system much more difficult and makes deterministic performance more difficult to achieve. Windows avoids paging of high memory as well.
4. Process Fork and Shared Memory
5. PSE36 RAM Disk
Technical Issues with Large Memory Support in IA32
Memory Sharing and Inter-Process Communications
To apply access and security controls, it is necessary to greatly increase the bookkeeping required of the operating system memory manager as well as the API set the application developer must use. This negatively impacts the high performance possible using very fast remap operations. It is also important to remember that IPC/memory sharing may still take place between two processes' virtual address spaces in any case, regardless of the physically mapped memory each may be using.
During a remap operation, it is necessary to ensure that all processors have valid logical-to-physical mapping on chip. Therefore, remap operations require a TLB shoot-down, because the logical-to-physical association is invalidated by the remap (where "logical" = the application/process view of memory).
There is a performance impact while the processor (or processors) reload the TLB. All operating systems have this issue, and in the case of PAE memory support, they ameliorate the issue in different ways:
Kernel and memory organization
One difference between operating systems is whether memory allocations are dynamic:
Given the method with which PCI addresses memory beyond 32 bits, there is a failure mode that is subtle. Any I/O range that "spans" across two 4-GB regions must be treated specially. If not, the address range will be correctly decoded for only one part of the transfer and the remaining part will be transposed to an incorrect memory location. This will corrupt memory and will crash the system, crash the application, or silently corrupt data at that location. Applications cannot prevent this because they are only presented virtual addresses and have no visibility to the physical level. All operating systems that use PAE face this problem, but some do not explicitly prevent this from occurring and instead depend on the device driver to take the correct actions.
Windows, however, explicitly prevents this problem. When an I/O range spans in this fashion, Windows returns two separate addresses and ranges to the device and driver. The final special case is the first transition from 4 GB to beyond. No DAC is required for the region below 4 GB, but DAC is required for the rest of the transfer. Again, Windows returns two separate addresses and ranges in this case to prevent memory corruption.
Obviously, DAC or 64-bit adapters and drivers provide the best performance as no buffering of I/O occurs. This buffering is required, however, whenever the adapter and driver cannot utilize more than 32 bits of address information. All operating systems that utilize PAE mode addressing support this "double buffering" in some fashion, as a backward compatibility feature. This buffering does have a performance penalty that is dependent on several factors:
As the physical memory increases, the relative amount of I/O addresses beyond 32 bits also increases in relation to those addresses below 32 bits. In most cases, the operating system transparently provides double buffering, although some Unix variants do not provide any assistance in this function and require any 32-bit devices and drivers to manage their own double buffering routines and allocations.
Several assumptions and shortcuts that could previously be used safely do not apply. In general, these fall in to three categories:
PAE mode can be enabled on Windows XP SP2, Windows Server 2003 SP1 and later versions of Windows to support hardware-enforced DEP. However, many device drivers designed for these systems may not have been tested on system configurations with PAE enabled. In order to limit the impact to device driver compatibility, changes to the hardware abstraction layer (HAL) were made to Windows XP SP2 and Windows Server 2003 SP1 Standard Edition to limit physical address space to 4 GB. Driver developers are encouraged to read about DEP.
Although this allows the obvious benefits of virtual memory, the downside is the performance impact on applications that have one or more of the following characteristics:
Finally, paging support typically comes at the expense of increasing the API set and slowing development and version migration.
The proliferation of proprietary APIs--some of which are tied directly to the processor architecture (kernel level)--makes porting applications from one Unix variant to another expensive, time-consuming, and a constant struggle to balance costs versus performance optimization. Windows provides an API set which is simple, fast, and completely portable between 32-bit and 64-bit hardware platforms, requiring only a recompile in order to function.
The issue with using varying page sizes for applications is related to additional application complexity required to function correctly with differing memory allocation sizes, as well as subtle effects related to the underlying assumptions that almost all applications have with page size. Although research shows a small class of applications can benefit from larger page sizes (2 MB or 4 MB), because each TLB entry spans a greater address range, the general rule is applications don't benefit from larger page sizes.
Windows and PAE
For more information about PAE and Windows, including guidelines for developers, see PAE Memory and Windows.