Common Logging File System

Article
06/24/2008

The System.IO.Log namespace defines an interface for logging to a record-oriented sequential I/O system. Using the classes from this namespace, you can implement your own diagnostic logging or transaction processing system. The namepsace also provides an implementation of this interface that uses a simple file-based log and an alternative implementation that uses the Common Log File System (CLFS) provided by ws2003r2 and Windows Vista.

System.IO.Log Namespace

The System.IO.Log namespace defines an interface for logging to a record-oriented sequential I/O system. Implementations of this interface can be used to read and write log records. When log records are appended to such an implementation, each one is given a unique sequence number. Sequence numbers are strictly monotonically increasing within a given record sequence, and numbers from different record sequences are not comparable. Sequence numbers are represented by the SequenceNumber structure. In addition, the record sequence provides a mechanism for reserving space in the underlying storage. You can take advantage of this reservation mechanism to ensure that necessary space exists for future log records.

Two different implementations of this interface are provided by the FileRecordSequence and LogRecordSequence classes. The FileRecordSequence is a record sequence based on a single log file in the file system.

The LogRecordSequence class, on the other hand, provides an implementation of the record sequence interface on top of a Common Log File System (CLFS) log. For more information on this implementation, see the "System.IO.Log Abstraction" section.

CLFS

CLFS provides a high-performance, general-purpose file log service that dedicated client applications can use and multiple clients can share to optimize log access.

Log architecture

The CLFS is an ARIES log manager. It saves log records in a sequential order, and can ensure that the flushed log records are preserved—even after a system failure. You can use CLFS to manage logs and set application policy. CLFS uses the following abstractions to implement logging:

A record is a unit of data that a client writes.
A physical log is a physical set of files and attributes that store one or many log streams. A log stream is a sequence of log records that a client logs. It is like a physical log. A dedicated log has only one, unnamed, stream. A multiplexed log has one or more named streams, and you can create more streams for a multiplexed log in the future. A client can log has one or multiple log streams in any type of log. However, after a log is created it cannot be converted from one type to the other.

A multiplex log represents a contractual agreement between two applications or subsystems that share the same log. Each stream in a multiplexed log appears to its client owner as if it were stored in a dedicated log. The primary advantage of using a multiplexed log is that system I/O costs are shared across multiple streams, so system performance may be better than if several dedicated logs are used. Records and log blocks in multiplexed logs are usually written to the same disk cylinder, which minimizes seeks and reduces I/O latency.

Log file access can either be directed to a local disk or to disks on remote systems via internal client-server support. Within a cluster, log files can fail over to another system using standard OS mechanisms. All writes to the log are buffered at the client until a flush or new buffer is required. When possible, data is written directly to disk from the client buffers without copying. Reads are cached to reduce disk access during recovery operations, backup, or bursts of transaction aborts.

The physical log is actually stored as a "base log file" (blf) which holds the metadata and any number of container files (or streams within a file). You can define where you want your log to be created, and create the log containers directly. You need to add at least two containers for the log to be usable. However, log policies can be set without any containers.

System.IO Log Abstraction

The LogRecordSequence class provides an implementation of the record sequence interface on top of a Common Log File System (CLFS) log. In addition to the standard record-oriented features, it provides a policy model for avoiding log-full conditions, and multiplexing of clients on the same physical file. It works with the LogStore class which provides an interface for directly manipulating and managing a CLFS log file. The relationship between the LogStore class and the LogRecordSequence class is similar to the relationship between a disk file and a FileStream object. The disk file provides the concrete storage, and has attributes such as length and last access time; while the FileStream object provides a view on the file that can be used to read from it and write to it. Similarly, the LogStore class has attributes such as a policy and a collection of disk extents; and the LogRecordSequence class provides a record-oriented mechanism for reading and writing data.

Unlike the file record sequence represented by the FileRecordSequence class, a LogStore instance stores its data in a collection of disk extents, represented by LogExtent instances. The extents in a given LogStore instance are all of uniform size, and space is added to and removed from a LogStore instance in extent increments.

The actual log records are represented by instances of the LogRecord class.

Policy

A LogStore instance can have policies associated with it. These are represented by LogPolicy instances. A policy dictates rules that the log will attempt to follow, such as maximum number of extents and minimum size, and instructions on growing or shrinking the LogStore under certain conditions. In addition, you can specify whether a LogStore instance is archivable or non-archivable.

Policies are set per log and are volatile, which means that once every handle to the log is closed, the policy no longer exists.

Security

A log is protected by standard NTFS ACL (Access Control List). It is not recommended to store a log on a FAT drive, as no protection is provided.