Updated December 2007
Revised February 2008
Summary: Because messaging is such a fundamental feature for many application architectures, poor choices can have disastrous consequences—affecting performance, scalability, availability, and, ultimately, user acceptance and the financials. In this brief article, we will focus therefore on the kind of questions that you should be asking when navigating this minefield. (10 printed pages)
What Is a Messaging Standard?
Are Messaging Standards Relevant/Important?
How Much Should I Make My Architecture Dependent on a Messaging Standard?
What Factors Affect the Longevity of a Messaging Standard?
What Areas Are Poorly Served by Standards?
What Gotchas Must I Watch Out For?
Should I Care What Goes On Behind a Particular Messaging Interface?
What About the Future?
Pick up any book on software architecture today, and you cannot help but notice the number of competing messaging technologies. With the emergence of the Internet and, subsequently, the Web, messaging now indirectly affects business efficiency and competitiveness to such a level that it has become a mission-critical part of software architecture. Given that there is so much at stake, the creation of standards—although often done on a voluntary basis—is rarely achieved without the influence of corporate agendas and individual ego; some would argue that these drivers are fundamental to getting anything done, actually. Now, new application paradigms, new consumer technologies, emerging security issues, new traffic patterns, platform and bandwidth resources, and costs are all driving messaging standards. We do not quite have the equivalent of the universal serial bus (USB) for the wide area; however, XML Web services appears to be by far the closest thing yet.
Because messaging is such a fundamental feature for many application architectures, poor choices can have disastrous consequences—affecting performance, scalability, availability, and, ultimately, user acceptance and the financials. In this brief article, we will focus therefore on the kind of questions that you should be asking when navigating this minefield. Hopefully, these apparently simple questions will expose subtleties that should inform your judgment on:
· How to analyze the merits of a standard objectively.
· How to anticipate the kind of issues that you might face during implementation.
· What the true costs are. Let's start with a simple question:
It should be clear what a messaging standard is—right? Well, the first issue here is that messaging means different things to different people, and so, unfortunately, does the word standard. Most of us naturally will think about networking when discussing messaging standards; however, messaging covers anything from a local platform-specific application-programming interface (API) down to a low-level communications interface such as NDIS. Other examples of messaging interfaces include POSIX threads, NetBIOS, RMI, CORBA, DCOM, XML Web services, Microsoft Windows' Winsock, UNIX's Berkeley sockets, and open data interface (ODI).
The term message itself refers to any unit of information that has to be moved. A message might be composed of just application data, or data encapsulated in your own messaging layer (if, for example, the interface to which you are writing does not support all of the functions that you need). It is useful to note that one layer's message is another layer's data. In practice, messages have been called many things, including packets, frames, events, or fancier names like protocol data units (PDUs).
The term standard also is not clear-cut and is widely interpreted in practice. There are true open international standards, de facto standards, and proprietary standards. True standards are usually defined by international standards bodies, or broad consortiums where "open" means that the standard has been forged by a community of interested parties, and these specifications should be available to anybody. The Internet Engineering Task Force (IETF) is perhaps the most open standards body in this area, because practically anyone can contribute, and all standards—even drafts—are available online at no cost. (Note that some organizations do charge a fee for their standards, and, while one can understand that resources do need support, I personally think that this inhibits the adoption process and the spread of knowledge.) De facto standards have become standards typically by virtue of their momentum, installed base, or the absence of any other useful alternative.
For example, the FIX protocol started life in 1992 as a bilateral communications framework for equity trading between Fidelity Investments and Salomon Brothers, and has since become the de facto messaging standard for pre-trade and trade communications within the equity markets.
How truly "open" and specified a de facto standard is might be highly variable and, therefore, could be open to interpretation. For example, Syslog is documented for informational use by the IETF, and several implementations have significant differences, requiring dedicated parsers.
Proprietary standards are normally vendor-specific. There might be full or no documentation available. There also might be license fees to pay. If the installed base is large enough, these standards nevertheless might become very important, and then often become de facto standards by virtue of their ubiquity.
The RTF specification is an interesting example of a widely used document-encoding standard defined by Microsoft. While strictly speaking it is a proprietary standard, it has become a de facto standard format for document exchange.
If you are new to this area, the maze of choices might seem bewildering. Even where the choices are limited or mandated, there are still many factors to consider. The first questions you must consider, therefore, are who is behind the standard, and how open is it, really? You must consider also whether there are licensing costs or associated usage restrictions, and how stable and complete it is. All of these issues can have a major impact on your architecture, implementation, and cost.
A Google search can give you a feel quickly for how much debate there is with a particular standard. For example, try typing "Web service standards problems"; see how many hits you get (clue: millions).
Standards are critical in promoting interoperability, consistency, and clarity when exchanging information between third parties. By adopting the right standards and using them in an appropriate context, you inevitably will save resources and ultimately contain risk and cost—leaving you to focus on organizational or business objectives. This is one area in which the reuse paradigm is best exemplified. In some domains—such as telecommunications, security, and banking—standards are mission-critical.
For some communities, messaging is so important that they cannot wait for established bodies to catch up. They might form their own consortiums to deliver an agreed method for communications (for example, FIX). Some of these "standards" might not have all the panache of international standards, and might look pretty crude in comparison. The key point is that they are made available and are agreed-upon methods for communicating in a particular domain.
In the security domain, Syslog has acted as the de facto standard for event-message communication for many years. Because it was only documented as an informational standard for BSD Unix, it was open to interpretation—leading to many different flavors, each requiring a different event parser. There is now an IETF working group for intrusion-detection message formats (IDMF), working to finalize Syslog as a true standard; however, neither is widely adopted.
Consider the context of the communications path. If your message peers are both internal to your system and only are ever required to perform simple operations, there might be little point in devoting months implementing a heavyweight messaging layer if a simpler one will do. Standards carry most weight where integration is required.
In the 1980s, the ISO OSI standards had significant momentum in Europe, especially for government and defense. Things often got a little strange in commercial environments, where OSI was used for backbone communications, even though expensive gateways were being used to translate OSI back to TCP/IP at every access point (that is, OSI was being used to talk to itself, and implementers were charging extra for the privilege).
Engineers can become passionate advocates for a particular technology, and messaging is no exception. There is a real danger of believing that the latest standard is the definitive solution, but history shows that this is rarely the case. Most software, however well-written, is ultimately disposable, and messaging standards change—often, more quickly than anticipated. Making your architecture dependent on any standards-based technology—instead of reliant on it—demonstrates a lack of foresight, and often has serious consequences later on, both technical and financial. Some flexibility must be designed in, from the start.
If we consider the last decade alone, we have seen major shifts from socket-style point-to-point connectivity, multicasting, remote objects, and publish-subscribe to grid and peer-to-peer computing. This is not so much an evolution of ideas (many of these are reinventions of older ideas) as it is an expansion of options.
Predicting the influence of a standard over time is not especially intuitive; there are many factors involved—some subjective, some conflicting, and some "unknown." Standards cannot exist in isolation from the sophisticated dynamics and realities of the environment in which they are to be deployed. Some standards try to solve so many problems that they end up achieving nothing particularly well. Standards can go out of favor quickly if environments change, and aggressive marketing can switch allegiances where there are competing standards. In extreme cases in which there is conflict during the standardization process, standards can be so heavily compromised at inception that they are of little value; or, even worse, the result is the creation of multiple competing standards. This usually ends in tears and significant expense for one camp. If you jump the gun and back the wrong standard, be aware that you could be leading your organization into a very expensive dead end.
Recent examples of dead-end standards include standards for information security and Web services, and various wireless standards. As recently as June 2006, the IEEE 802.20 working group was suspended because of the commercial infighting taking place. Intel, Kyocera, Motorola, Qualcomm, and other giants with high stakes in the market all have representatives on the working group, which at the time numbered around 175 people.
Perhaps the most appropriate answer is that the fittest standards tend to survive longest (although this does not necessarily mean the best technically). Standards that have the strongest chances of success will exhibit common features:
· Broad participation from many interested parties (vendors, academia)
· Extensive peer review
· Relatively simple to understand
· Accessible and well-articulated
· Appropriate use of resources (memory, CPU, bandwidth, and so on)
· Simple to implement
· Significant momentum from respected players
· Ideally, a vehicle for promoting rapid expansion of the installed base
A perfect example is the venerable TCP/IP standards, standardized by the IETF nearly 30 years ago and still serving as the basis for practically all remote messaging that is used today. We easily forget that, at the time, there were many competing standards (ISO OSI, Novell NetWare, and Banyan Vines, among them). We also forget that these protocols were developed at a time when platform and bandwidth resources were very scarce. TCP/IP was relatively simple and inexpensive to implement, was shipped free with many UNIX platforms, and was rapidly adopted by many U.S. appliance vendors. While the battle lingered on for several years, the relative simplicity and efficiency of this standard, plus its widespread installed base, generated such momentum that it eventually dominated the field completely. Furthermore, those initial resource constraints are once again important within mobile and PDA communications in which efficiency is paramount for battery life and bandwidth costs.
It is worth noting that TCP/IP is far from perfect. There are major issues with congestion control and address space, and there have been many "patches" to improve performance over the past 30 years. Nevertheless, this is perhaps the prime example of a fittest survivor in the evolution of communication standards, alongside Ethernet.
There are many examples of standards that have not achieved projected targets, including OSI and asynchronous transfer mode (ATM). In the 1980s, the TCP/IP-versus-OSI debate raged for several years until the situation today, where what remains of OSI is primarily the seven-layer conceptual model, plus a few specialized domains of dominance. In the commercial world, OSI does not really exist. Similarly, in the 1990s, the new ATM seemed assured of the dominant position for all backbone and client-server media communications. Issues such as cost and the resistance of the incumbent installed base meant that ATM never seriously encroached into the Ethernet space.
In more recent times, standards such as CORBA and DCOM have been embraced in particular industries (such as finance), but are increasingly falling out of favor with the introduction of XML Web services. Even though these technologies might offer very sophisticated features, the cost to deploy and maintain these infrastructures often becomes a significant burden when organizations are grappling with a dynamic environment and periods of severe cost constraint.
Where there is rapid change, or complexity, or uncertainty in a particular industry sector, there are always deficiencies. For example:
· There still is no genuine consensus on standardizing security-event messages. Security is one area in which you really need clarity and agreement, because this lack of commonality results in information loss, compromised security, and increased costs (such as where events have to be normalized or parsed differently).
· There still are issues mapping applications to the transport layer, with many architects choosing to write proprietary messaging protocols to fill this void. One industry veteran, Marshall T. Rose, got so fed up with reinventing the wheel that he devised a new framework called Blocks Extensible Exchange Protocol (BEEP) and proposed new IETF standards.
Unfortunately, as an architect, you must decide whether a standard is completed enough for you to adopt it, and how you will address any deficiencies over time. That might require you to implement proprietary features over the standard until such features are fully crystallized.
There are many decisions that must be made in how actually to implement to a standard; that is, there is still a lot of art involved in this process. Mistakes in this critical area of architecture can be very costly, and issues that you must consider are as follows:
· What real integration and business benefits are likely, and what are the risks?
· How stable and complete is the standard? Can you migrate from proprietary features as the standard matures?
· What is your expected resource utilization, such as platform footprint, network latencies, and so on, and how will it determine your ability to scale (and, ultimately, your cost)?
· What infrastructure support is required (things such as ports open on firewalls, routing, how will code be deployed, updated, or managed)?
· What might be your hidden costs to support robustness, complexity, training, implementation, tools, and maintenance?
However good your judgment is, things can and do change; therefore, you must balance flexibility with making optimum use of the available interfaces. Be wary of using exotic features that might lock you in, unless they are really justified. Adopting the wrong standard can be an expensive mistake.
In the mid 1980s, there were competing messaging standards for network management, essentially split between the IEEE 802.1 and the IETF's SNMP (ignoring the alternative ISO OSI management standards). Several companies chose the IEEE path. At that time, the best choice was far from obvious, but SNMP eventually succeeded. For those companies that adopted the IEEE standard, their potential for substantial market leadership was lost at considerable expense.
Judging by the knowledge exhibited by the majority of candidates just out of university, it would seem that application programmers are taught to ignore underlying messaging operations. This, in my view, is a big mistake. While considerable effort has been spent in abstracting the "plumbing" from the application programmer (clearly not a bad thing), you still have a responsibility to understand, firstly, what is going to happen to your messages when they get placed on the network and, secondly, what resources they will consume. You must invest some time to make the most efficient use of that interface and to avoid expensive mistakes in not understanding the impact and requirements of the communications infrastructure.
A few years back, I sat in a development meeting in which we discussed how we could best deliver messages between remote agents. One of our youngest and brightest engineers made a passionate case for IP multicasting. What he failed to appreciate was the fact that multicasting placed a major burden on routing resources at the time, making it prohibitively expensive to deploy. Often, the decisions that we make on messaging—even though they might have the best technical merit—can have a disastrous impact on the overall cost of ownership.
The selection and use of a messaging interface can affect scalability, usability, and (ultimately) cost significantly. Even when an interface is well-specified, there often are major decisions about how best to use it in the context of the application. For example, should you use blocking or asynchronous modes, threads, or events? Poor decisions can lead to a compromised solution that ultimately might be scrapped—often, at considerable expense.
In the last few years, there has been an explosion in new messaging paradigms—driven largely by the Web, the mobile-communications industry, and emerging entertainment and media services. Instant real-time messaging between everyone and everything is becoming pervasive—from your game console, your car, home musical equipment, your MP3 player, mobile phone, your washing machine, and, ultimately, you. The constant battle between resource availability and a richer user experience will continue, as the boundary between the real and the virtual begin to blur. If we accept that resources are being made available at an exponential rate of growth, this boundary eventually will disappear—all supported by messaging standards.
Engineers understandably want to work on the very latest technology; so, exercising good judgment and restraint in this area is a skill in itself. For most commercial applications, you must save bleeding-edge technologies for research and experiments; it is never a good idea to bet the company on them. Ultimately, your responsibility is to design an application that is fit-for-purpose, within constraints that often conflict or are uncertain. Inevitably, you must arrive at solutions that:
· Meet basic user requirements for functionality and performance.
· Operate predictably under worst-case conditions today, and will scale to meet predicted future demands.
· Incorporate features to help differentiate from competition.
· Are cost-effective to implement and deploy (identifying all costs, including training, development, maintenance, update, configuration and management, and external licensing and maintenance).
· Offer a return on investment (ROI) with tangible benefits in relation to operational efficiency, productivity, cost control, and risk reduction.
· Integrate well with legacy systems and support the core business objectives.
· Accommodate flexibility for potential obsolescence and change.
· Ensure compliance with associated regulatory or organizational standards.
Ultimately, this is about making pragmatic decisions that maintain an acceptable risk profile for your organization. The messaging layer is just one—albeit important—component in the overall software architecture and, ideally, should be treated as a commodity.
In closing, we should all spare a thought for all of those individuals who volunteered years of their lives developing these standards that we take for granted.
· FIX Protocol Organization—The Financial Information eXchange (FIX) Protocol is a messaging standard developed for the real-time electronic exchange of securities transactions. FIX is a public-domain specification that is owned and maintained by FIX Protocol, Ltd.
· Kurzweil, Ray. The Singularity Is Near: When Humans Transcend Biology. New York, NY: Viking, 2005.
· Word 2007: Rich Text Format (RTF) Specification, version 1.9. February 2007. Microsoft Corporation, Redmond, WA. [Cited January 10, 2007.]
· Rose, Marshall T. "The Blocks Extensible Exchange Protocol Core." RFC3080, March 2001. Internet Engineering Task Force.
· Rose, Marshall T. "Mapping the BEEP Core onto TCP." RFC3081, March 2001. Internet Engineering Task Force.
· Lonvick, Chris. "The BSD Syslog Protocol (Informational)." RFC3164, August 2001. Internet Engineering Task Force.
· Postel, J. "Internet Control Message Protocol."
· RFC792, September 1981. Internet Engineering Task Force.
· Postel, J. "Transmission Control Protocol: DARPA Internet Program Protocol Specification." RFC793, September 1981. Internet Engineering Task Force.
· Rose, Marshall T. BEEP: The Definitive Guide. Sebastopol, CA: O'Reilly, 2002.
There are many bodies in the standards arena, notable examples of which include:
· IEEE Standards Association—A leading voluntary organization for standardizing physical-level network communications, such as LAN wired and wireless communications.
· IETF (Internet Engineering Task Force)—A voluntary professional organization that has developed most of the standards that are used today in Internet communications.
· ISO (International Organization for Standardization)—Defines international standards, including the seven-layer OSI model.
· ITU (International Telecommunication Union)—Standards bodies for telecommunications protocols, including PSTN, radio, and so forth; also, compression standards for modems. Previously known as the CCITT.
· OASIS—A nonprofit, international consortium that creates interoperable industry specifications based on public standards, such as XML and SGML.
· W3C (World Wide Web Consortium)—Develops interoperable Web technologies: specifications, guidelines, software, and tools.
About the author
Tony Kenyon has worked as an engineer in IT since the early 1980s, spanning emerging fields in data communications, mobile technologies, security, and Web services. A graduate of the University of Leicester, England, he is a member of the Institute of Analysts and Programmers, ISAS, and the IEEE, and holds a CISSP.
This article was published in Skyscrapr, an online resource provided by Microsoft. To learn more about architecture and the architectural perspective, please visit www.skyscrapr.net.