Design Patterns
Simplify Distributed System Design Using the Command Pattern, MSMQ, and .NET
Brad King
This article discusses: - Distributed computing in a homogeneous network
- Queuing for balanced resource utilization
- Achieving good reliability and scalability
| This article uses the following technologies:
Design Patterns, MSMQ, .NET, C#, SOA
|
Code download available at:CommandPattern.exe(129 KB)
Browse the Code Online
Contents
These days, discussions about distributed computing and the Microsoft® .NET Framework frequently focus on service-oriented architecture (SOA). But when you want to perform processing inside your local network, the Microsoft Patterns & Practices Enterprise Solution Pattern suggests that object-based collaboration may be more appropriate than service-based collaboration. In the local environment, systems generally share a common platform, allowing you to optimize in ways you can't on widely dispersed systems that span various operating systems and development environments.
While problems generally associated with distributed object solutions still exist, such as object lifetime control, reliability, and scalability, you can minimize these by developing a hybrid solution that uses Windows® services and Microsoft Message Queuing (MSMQ) to implement a queued Command design pattern. This solution is described in the seminal book Design Patterns by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides (Addison-Wesley, 1995). In this article I'll discuss how a queued command system based on that pattern can be built using C#, the .NET Framework, and MSMQ.
Placing this in the context of enterprise application design, the Command pattern supplements the patterns specified in the guidelines discussed in Microsoft Patterns and Practices . I highly recommend reading over these guidelines for an overall view of patterns in the enterprise, but this article doesn't rely on any specific information from that text.
The distributed command system discussed here is the combination of reliable messaging, services, and the Command pattern. While most often seen as a design pattern for implementing desktop application menu systems, the Command pattern offers advantages for distributed systems when combined with a robust messaging infrastructure like MSMQ. The pattern itself gives structure to the programming environment. It provides a single way to create work and to execute it, while MSMQ provides the infrastructure to distribute the work to near links on the network. Using this pattern, the application developer can work with all aspects of the system from within the .NET Framework, which makes it an extremely efficient working environment. To drill down deeper, let's first look at the Command pattern and the part each participant plays.
Command Pattern Refresher
According to the book Design Patterns, the Command design pattern "encapsulates a request as an object, thereby letting you parameterize clients with different requests, queue or log requests, and support undoable operations." Notice the part about queuing—that's important to us. The Command pattern has the participants shown in Figure 1.
Figure 1 Pattern and Participants
| Class | Description |
|---|
| Command | Abstract base class for all Command objects |
| Concrete command | Any number of classes derived from Command |
| Client | The creator of a command |
| Invoker | Executes a command |
| Receiver | Any class that is acted on by a command |
In its simplest form, the Command base class defines a single method called Execute, which takes no arguments and returns nothing. Concrete Commands derive from Command and implement Execute by calling into one or more Receiver classes to ultimately achieve whatever work is necessary. The contents of Execute are fairly open. You may in fact have a completely self-contained Execute function that has no external dependencies. Figure 2 shows the relationships between the participants.
.gif)
Figure 2 Command Relationships
Pattern purists will recognize that there are differences from the original version found in Design Patterns. First of all, the client doesn't hold an instance of the receiver object because it's not feasible (or desirable) to do this across the network in our queued design. The same is true for the relationship between the command itself and the receiver. A client never gives a command an instance pointer to a receiver. Instead, a command can create whatever objects it needs to use directly in its Execute function. I don't believe the intent of the original pattern is lost here, just modified to work in a distributed environment.
The implementation in this article uses the ICommand interface as the base for all concrete commands, although an abstract base class would serve just as well. Figure 3 shows a very simple concrete command class.
Figure 3 Simple Command
public interface ICommand { void Execute(); } public class SimpleCommand : ICommand { public SimpleCommand() { } public void Execute() { Console.WriteLine("SimpleCommand.Execute complete"); } }
To use this class, a client simply creates a new SimpleCommand object and passes it off to an invoker object. The invoker has the incredibly exciting task of calling ICommand.Execute over and over again on anything that comes its way. While based on simple inheritance, this is one of the coolest things about the Command pattern: there is only one way to execute any command. In reality, useful commands usually carry some state with them. Maybe it's an identity for an employee in a SQL database or the complete recipe for making cookie dough, both in a collection of strings. Whatever it is, the pattern sequence is the same for all commands. The client creates a command and sets its properties, and the invoker tells the command to execute. The command then acts on any number of receivers to accomplish its task. A concrete command class will often implement a parameterized constructor to accept the necessary arguments to set some or all of its properties, as shown in Figure 4.
Figure 4 Command with State
public class ReportRenderCommand : ICommand { protected int ReportID; public ReportRenderCommand(int reportID) { ReportID = reportID; } public void Execute() { ReportEngine r = new ReportEngine(); r.Render(ReportID); } }
A client can create an instance of this command, and invoke it to render report ID 1000, like this:
ReportRenderingCommand r = new ReportRenderingCommand(1000); r.Execute();
In this case, the client and invoker are one and the same. The receiver is a ReportEngine object, which is created by the command in the Execute function and is passed the report ID to render from the command internal state. Commands themselves are a topic that could take up the rest of this article. Commands can support Undo operations by implementing an Unexecute function, or multiple commands can be aggregated to form a macro command. You can exploit those variations in the distributed architecture as well, but I won't discuss that here.
Distributing Commands
The 1000-foot view of the combination of Commands, Services, and Messaging goes something like the following. Commands are implemented as classes that encapsulate requests, as we've seen. The Client manages the creation of commands and puts them into a queue. MSMQ delivers the commands to an invoker processing service somewhere on the network. The invoker reads the command from the queue and calls its Execute method. The invoker is implemented as a service, treats all commands the same, and doesn't need to know anything about a command to invoke its operation. Commands are autonomous and can be added with no code changes to the Invoker service.
The primary goal of queuing commands is to invoke the command on a different server than the client, although it could just be a separate process or application domain on the same server. If you're going to separate the client from the invoker, there are some things to consider up front. The exact same command types must exist on both the sending and receiving ends of the queue. If you add a new command or change an existing command, you must proliferate the changes to all clients and invokers.
Let's look at the messaging side of things. MSMQ is a reliable channel for moving almost anything from one place to another on the network, including objects. Queuing, by definition, smooths out demand peaks in your application by buffering a unit of work until the processing can catch up at a known maximum rate. You can control the rate of work consumption by adding threads or servers on the receiving end of the queues. This is not very easy to do with systems that don't expose dynamic controls on threading like Web services, asynchronous delegates, or .NET Remoting. Consider the report-rendering command we talked about earlier.
Let's say you've written a report-scheduling application for your company, which has thousands of reports. When a user is setting up a report to run unattended, the default time to execute a report is 2 A.M. So guess what happens? Most users schedule their reports to run at 2 A.M. because it takes effort to change a schedule from the default time, and reasonably enough, 2 A.M. sounds just fine to most people. Only the engineers in the crowd will actually stagger their report scheduling. To accomplish this workload, you either need a queuing system or a lot of horsepower available in those early morning hours. Queuing is a better use of resources since during the remainder of the day the reporting servers will be doing a very small percentage of the peak demand that occurs at 2 A.M. MSMQ provides this vital infrastructure for you, with a nice .NET Framework SDK on top of it.
You can accomplish queuing work in many different ways. If you are using SOA, you might generate a SOAP message and send it to a queue where you know a service that can process that message is listening. As new message types are added, the service that processes them also changes because the message contains only state and no knowledge of what should be done with that state. This is absolutely essential when integrating disparate systems in the enterprise, but not when you have complete control of the platform. Queuing command objects takes the message state and adds the intelligence to know what to do with the state once it reaches its destination.
The participants in the distributed command architecture are conceptually the same as in the original design pattern. However, the functionality is definitely different for client and invoker, and the difference is not transparent to those participants. This is by design. If a command should be distributed, the client application developer probably needs to know about it and write code with that in mind. Location transparency isn't a design goal for distributed commands. A conceptual view of a queued command system is shown in Figure 5.
.gif)
Figure 5 Distributed Command Participants
Let's look at each Command pattern participant and how it changes in the distributed world compared with local invocation. Let's also look at the new participant, MSMQ.
Messaging Infrastructure
I won't go into a complete discussion on installing and administering MSMQ since there is ample information in articles and SDK docs dedicated to that subject. To the developer, MSMQ is just infrastructure and only needs to be touched outside of the application for queue maintenance. As a designer, you can think of a message queue as actually playing two roles: a transport channel and an addressable endpoint. A queue is a tangible thing that lives on a particular server on your network. The collection of available public queues is managed by a domain controller which stores queue locations in Active Directory® (MSMQ 2.0 and higher). Private queues are not listed in Active Directory, but can still be addressed by other machines on the network that know its location.
MSMQ is an installation option in the control panel under the Add/Remove Windows Components dialog. In the details, you can select to install Active Directory integration, MSMQ Triggers (a COM-based service technology), and HTTP support (MSMQ 3.0 only, on Windows Server™ 2003 and Windows XP), along with the basic queuing functionality. You will want to install the independent client (the default), not the dependent client. Avoid unchecking the Local Storage option under the Common details since doing so results in a dependent client install. Dependent clients have no store-and-forward functionality, and won't be supported in future releases of messaging. The MSMQ independent client runs as a Windows service. You should coordinate with your technical operations team if MSMQ is not already installed on your servers since you will want these requirements integrated with your server setup documentation and deployment procedures.
You can create public or private queues manually in the Computer Management tool, Server Explorer in Visual Studio® .NET, or programmatically. The Computer Management tool provides the most functionality of the three.
As a transport, MSMQ is feature-rich out of the box. You get store-and-forward technology for offline clients. You can set priorities on messages so more immediate commands can jump in front of longer-running tasks in the queue if necessary. Messages can be sent as "recoverable," meaning they are written to disk when they enter the queue rather than just living in memory. That way, if the machine crashes while there are messages in the queue, the work won't be lost. Queues can be transactional, so the read and write operations can be rolled back if aborted. There are many other features as well, but the main thing is that it's reliable, fast, and works well with the .NET Framework.
As a service endpoint, message queues can be addressed in several different ways. You can create a queue with a location-independent label, and clients can open the queue knowing only the label name. A complete path to a machine\QueueName can also be used or you can use a direct link to the queue, bypassing Active Directory altogether. With System.Messaging, you can easily switch addressing modes by changing string prefixes on the queue locator string you provide in the System.Messaging.MessageQueue constructor. For example, the address strings in Figure 6 will all locate a public queue called TestQueue on the computer MyMachine.
Figure 6 Address Modes
| Address Mode | C# String |
|---|
| Path | @"MyMachine\TestQueue" |
| Format name | "FormatName:Direct=OS:MyMachine\TestQueue" |
| Format name | @"FormatName:Public=9423F746-4D1B-4DFD-8E3C-FF96C92E6254" |
| Label | @"Label:TestQueueLabel" |
While each address mode will locate the queue, there are big differences in how each mode resolves the string to a queue location. For example, a "Label:" address mode forces MSMQ to go to Active Directory for a lookup, but a "FormatName:" address does not. Both the Path and Label modes do a lookup on the domain controller to get the internal format name (a GUID) to open the queue. You can get the format name for a queue by looking at the ID shown in the queue properties window of the management tool. Be careful with Label mode as well, since a queue label doesn't have to be unique on the network. Each mode offers a performance-location transparency trade-off that you should consider when selecting a mode for your client and invoker applications to use. Queuing details like these are important to the invoker service and to the client, which is the entry point to the queue system.
The Client
The client in a queued command system has two primary responsibilities: to create a command object instance, including setting any internal state needed by that object, and to put the message into a queue. Creating a command object instance is no different in a distributed system. Queuing the command involves locating a queue, opening it, selecting the correct message formatter and any other queue options, and finally, sending it to the queuing system. While this might sound like some extra work for the client, it's really pretty easy thanks to the classes in the System.Messaging namespace. First you add a reference to System.Messaging. Here's a function from the client sample code that can queue any command that derives from ICommand, given a path to a queue:
private static void QueueCommand(ICommand c, string destinationQueue) { // open the queue MessageQueue mq = new MessageQueue(destinationQueue); // store message on disk at all intermediaries mq.DefaultPropertiesToSend.Recoverable = true; // set the formatter to Binary, default is XML mq.Formatter = new BinaryMessageFormatter(); // send the command object mq.Send(c, "Invoke Command"); mq.Close(); }
The destination queue must already exist when the QueueCommand function is called; otherwise a MessageQueueException will be thrown. The sample code in the download creates the local private queue for you on startup. Check out David Platt's article in the December 2003 issue of MSDN®Magazine (MSMQ and .NET: Send MSMQ Messages Securely Across the Internet with HTTP and SOAP)and the SDK docs referenced at the end of this article for more details on creating, locating, and opening queues with System.Messaging.
The client code uses the System.Messaging.BinaryMessageFormatter class to serialize objects into the body of a queue message. We want an exact graph of the command object's internals. The default formatter is XMLMessageFormatter, which is designed for loose coupling, so we have to change it anytime we create a new MessageQueue object.
While accuracy is important, most commands will actually serialize just fine with the default XML formatter, although there is a potential problem waiting for you on the other end of the queue. The XML formatter requires a list of all types that it can deserialize. That means anytime we add a command class, we have to modify the invoker code to add a new type to the XML formatter so it knows that this type is actually eligible for instantiation.
You can certainly use the XmlMessageFormatter if you want to tightly control which types of commands can be executed, but it's cool to be able to add new commands without changing the service code. There are also some types which will not serialize properly with XMLMessageFormatter. For example, you can't serialize a Hashtable or an object array containing other arrays of strong types like strings or integers. Binary formatting works great for this application, and there's no reason that you shouldn't use it.
Commands
Commands have two additional requirements when distributed by MSMQ. First, they must have a default constructor so the BinaryMessageFormatter can create a new object instance when the message is received. Second, they need to be marked with the [Serializable] attribute or you'll get a serialization exception when trying to send the object to the queue.
There is actually one more requirement, but it's related to how you factor and deploy your assemblies. Both the client and the invoker need to be able to find the same command objects at run time. Depending on how many applications will be queuing and invoking commands, you may find it necessary to put them into one or more strong-named assemblies and register them with the Global Assembly Cache (GAC), allowing them to be shared. This is really an application deployment decision on your part, and commands are quite happy running as "copy local" references within a single .NET Framework project.
An interesting twist on the distributed command is a generic or late-bound command. This command can invoke a static method on any assembly within the scope of the invoker. This is a lot like a delegate, but it shares the common command interface and thus can be queued and invoked like any other command. The late-bound command relies on the client having explicit knowledge about the assembly, class, and method it wants to invoke. The command Execute function uses reflection to execute the method on the receiver. This is a quick and dirty way to convert logic from local to distributed, without constantly rolling out a new command object. Figure 7 shows the late-bound command.
Figure 7 Late-Bound Command
[Serializable()] public class LateBoundCommand : ICommand { protected string AssemblyName; protected string ClassName; protected string MethodName; protected object[] Parameters; protected LateBoundCommand() { } public LateBoundCommand(string assemblyName, string className, string methodName) { AssemblyName = assemblyName; ClassName = className; MethodName = methodName; } // sets all parameter values public void SetParameters(params object[] myParameters) { Parameters = myParameters; } public virtual void Execute() { Console.WriteLine("LateBoundCommand.Execute starting"); Assembly a = Assembly.LoadWithPartialName(AssemblyName); Type TargetType = a.GetType(ClassName); MethodInfo TargetMethod = TargetType.GetMethod(MethodName); TargetMethod.Invoke(null, Parameters); Console.WriteLine("LateBoundCommand.Execute complete"); } }
The client creates the command by providing an assembly name, class name, and method name to be called in the command constructor. The client provides parameter values for the method using SetParameters on the command. This method takes a variable number of argument values as a param object and stores them in an object array to be transported with the command. The process of creating and sending a late-bound command is shown here and is also included in the sample project:
private static void QueueLateBoundCommand() { LateBoundCommand c = new LateBoundCommand("ReceiverSamples", "QueuedCommandSample.TestReceiver", "ActionStatic"); // the ActionStatic method takes 2 parameters c.SetParameters("Hello late bound", 101); QueueCommand(c, QueuePath); }
Note that the assembly name does not include the .dll extension. When the invoker calls Execute on this particular command instance, the TestReceiver.ActionStatic(string, int) method will be called, with arguments "Hello late bound" and 101, respectively. You should keep in mind that the receiver method has to be static in this implementation of the command since we are passing null as the instance argument to MethodInfo.Invoke in the command Execute function.
One final thing to remember is that commands must be less than 4MB in size when they are serialized because that's the size limit for an MSMQ message.
Invoker Service
A command has to be executed to be useful. The invoker is a simple but powerful service that listens for command messages on a certain queue and executes them. The invoker doesn't need to know anything about the command at all except that it is required to implement ICommand.Execute. The first thing an invoker must do is open a queue and tell the queue that it should use the binary formatter for deserialization:
MessageQueue mqReceive = new MessageQueue(QueuePath); mqReceive.Formatter = new BinaryMessageFormatter();
Once the queue is open, we can periodically poll it for new messages. This is a safe way for multiple invokers to read from the same queue, as shown in the following:
Message myMessage = mqReceive.Receive(TimeSpan.FromMilliseconds(500)); ICommand cReceived = (ICommand)myMessage.Body; cReceived.Execute();
The formatter returns the message body to us as type object, so we need to cast it to ICommand. All that's left to do is execute the command. If no message arrives in the queue within the time span specified to receive, or another service gets the message, the Receive function times out and a MessageQueueException with a code of IOTimeout is thrown, which you must basically ignore:
catch (MessageQueueException e) { if (e.MessageQueueErrorCode == MessageQueueErrorCode.IOTimeout) { // do nothing } else ••• }
Generally the invoker needs to be a Windows Service application or standalone executable, since it continuously monitors the queue for inbound messages. Service applications have the advantage of automatic startup and the service controller interface to start and stop the service on demand. Since message queuing is operating behind the scenes, an invoker service can be stopped while commands build up in the queue. Once the service is restarted, it will pick up right where it left off as long as the time to live for the message has not been exceeded, at which point the message expires and then disappears. You can control the maximum time a message can take to reach its destination, and the maximum time it can wait to be read from the queue using the Message properties TimeToReachQueue and TimeToBeReceived.
Scalability and Reliability
MSMQ easily supports multiple remote writers to the same queue. The key to scalability lies in the ability to expand the invoker queue network to multiple servers, and also in the mapping of your commands to the queues that form service endpoints. You could potentially run all commands through one queue and one invoker application (as the sample code provided with this article does), but that's unlikely in a production setting. An initial estimate of the quantities and types of commands you expect your application to dispatch will help you get started. Excessive overloading of a single invoker service makes the system hard to understand, but an invoker service per command type is probably too granular. Factoring by grouping similar commands is a good start, or your application may already have other logical groupings. This is something that takes several iterations to nail down.
The invoker is the workhorse of the system and has to scale to meet the throughput needs of the application. While queuing softens the demand peaks, the average rate of work consumed over time still has to exceed the rate of work sent or the system will be overrun. For a single server, additional threads can be added to the invoker service to increase throughput. Unfortunately, implementing threading limits in .NET isn't as simple as it could be. The ThreadPool class doesn't have a friendly property to set the maximum number of threads in the pool per process. The good news is that it's not terribly difficult to roll your own collections of threads. There are also open-source .NET thread pool classes available. Monitoring the number of items in each queue in Performance Monitor or Microsoft Operations Manager (MOM) will give you a good idea of how well the invokers are keeping up. The individual queue performance counters provided by MSMQ should be used for this.
Scaling the invoker out to other servers can be accomplished in a couple of ways. The easiest (but not the recommended) way to scale out a service endpoint is to simply add more servers running invoker services reading from a single physical queue. This configuration should be explicitly avoided, according to the MSMQ documentation, because it consumes too many resources to read from a queue remotely. However, as Ted Pattison described in his excellent book Programming Distributed Applications in COM+ and Visual Basic 6.0, 2nd Edition (Microsoft Press®, 2000), this is the optimal multi-server listener configuration to ensure no listener sits idle. This setup won't work for reading transactional queues. Remote transactional reads are just not supported by MSMQ, even in version 3.0.
There are ways to handle this problem, however. One solution is to inject a dispatcher service on one server which load-balances the messages from a central queue out to individual queues on each server running an invoker. This is described in the MSMQ technical article: "
Message Queuing: A Scalable, Highly Available Load-Balancing Solution".
Figure 8 shows a scalable distributed command architecture using the dispatcher concept.
.gif)
Figure 8 Scaling Invoker
The dispatcher becomes a virtual invoker service endpoint with a load-balanced queue farm behind it. The dispatcher must keep a list of those individual destination queues and apply a load-balancing algorithm to decide where to forward the commands. Alternatively, you can implement a request/response queue setup where the invokers ask the dispatchers for work. There is some good information on simulating remote transactional read semantics using an intermediary like the dispatcher in the article "
Microsoft Message Queuing Services (MSMQ) Tips".
Conclusion
The Command pattern specifies a set of interactions that you can apply to the development of distributed systems in .NET. Use this pattern in your application to implement a high-performance, extendable backbone connecting your near link nodes that run .NET-enabled operating systems to achieve optimal performance and rapid development. Distributed commands are a mix of classic object orientation, messaging, and service architecture that together fill a definite need in the distributed systems space.