Developing Distributed Services Today

Article
04/03/2007

Richard Turner
Program Manager
Microsoft Distributed Systems Group

July 2005

Summary: This paper provides guidance on how to best use Microsoft Web service and distributed systems technologies in order to build distributed, service-oriented systems using today's platform. The impact of future technologies such as "Indigo" is considered, and how Microsoft products such as BizTalk Server, SQL Server Notification Services, and Host Integration Server are positioned in this space. (12 printed pages)

Introduction
Distributed Systems
   Distributed Objects
   Services
Guidance – Building Services Today
   Guidance Summary
   Build Code Using .NET
   Build Web Services
   Keep Objects and Components Inside Your Services
   Use System.Messaging/MSMQ for Queued Messaging
The Future of Microsoft Distributed Systems Technologies
   A Brief Introduction to "Indigo"
Associated Products
   BizTalk Server
   SQL Server
   Host Integration Server
Summary
Appendix 1: References

Introduction

This paper will help you make the best choices when deciding how to build distributed systems today on the Microsoft platform.

Let's begin with a brief discussion of why distributed objects became very popular and why we at Microsoft are now adopting the principles encapsulated within the notion of "Service Orientation" to help deliver distributed systems more effectively in the future. I discuss how to build loosely coupled services using ASP.NET Web services (ASMX) and Web Services Enhancements (WSE), and then present guidance on when and where to most appropriately leverage COM+ Enterprise Services, System.Messaging, and NET Remoting. I close with a brief overview of the future Microsoft technology roadmap and guidance on how other Microsoft products such as BizTalk and SQL Notifications compose with our distributed systems platform.

Distributed Systems

Microsoft has delivered several generations of platforms for distributed computing. In supporting and enhancing these products, we have learned a lot (much of it from our customers) about what principles do and don't work over time when building distributed systems.

Distributed Objects

Object technology was designed as a local programming concept that assumed many things in order to make the technology work. For example, object technology originally assumed that objects would run within the same process and on the same platform (OS or runtime). We then began to stretch that object technology by building components that spanned multiple processes on a given machine (using COM and, more recently .NET). We then stretched this technology further still by providing mechanisms (COM+ and later, .NET Remoting) that enable components running on more than one machine to communicate. Many distributed systems have been built on our distributed component technologies. These systems can exhibit extremely high levels of performance, security, reliability, and availability.

However, while it is initially convenient to build systems using distributed components, this approach is often constrained by a number of factors, including:

Interoperability: Distributed component technologies today (regardless of vendor) rarely provide rich interoperability with other technologies or platforms.
Coupling & Autonomy: In systems built from distributed components, the components containing the data types and behaviors shared between callers and services must be installed on the server, as well as any callers of that server. This results in very tight coupling, which impacts:
- Versioning: Tightly coupled, component-based systems are notoriously difficult to change and adapt to changing business requirements. If a change is required, the server and all clients must be updated atomically in order to avoid breakage.
- Availability: Systems which are tightly coupled are often fragile and unable to gracefully handle failures
Transparency: Components which were not specifically designed to be distributed are unlikely to meet security, performance, scalability, and other requirements if they are distributed more broadly than they were designed to support

Therefore, while distributed component technologies are a very powerful way to construct tightly coupled portions of your systems, they are not the most appropriate technologies to use when building distributed systems that must be interoperable and able to be quickly adapted to changes in the business requirements.

Services

Microsoft has been providing distributed component technologies ever since the introduction of DCOM in 1993, and we have worked with thousands of customers, partners, and ISVs around the world. We have learned a great deal throughout this time in terms of how distributed systems should be built. Over the last several years we have been honing our approach to building distributed systems and have come to learn that in order to do so, a modified set of assumptions should be adopted:

Instead of assuming the use of objects and scoping a system to the capabilities of a chosen platform, we should instead assume that we have no control over the applications and systems that we interact with. We should not assume that systems beyond our own will be running on the same machine architecture, operating system, platform, or developed using the same tools. In the future, we will assume that the world beyond our borders is a hostile, unreliable, insecure place, but that we will choose to cooperate with other systems that honor our agreed contracts, schemas, and policies:

Contract: Specifies what a service looks like, what operations the service provides, what Message Exchange Patterns (MEPs) the service's operations support (simplex, duplex, or request-reply)
Schema: Defines the data structures exchanged between services and callers when invoking operations on the service
Policy: States transport-oriented requirements such as what types of security, reliability guarantees, and transaction support are required

These new assumptions led us to adopt and develop the notions that are encapsulated by the term "Service Orientation" (SO). SO encapsulates the principles and best practices necessary for building future distributed systems that avoid many of the problems inherent in today's distributed component technologies.

Guidance – Building Services Today

This guidance we present below is born from our extensive experience in delivering Microsoft distributed systems and comes after consultation with a large number of customers and partners. While this guidance is valid in most scenarios, there will be "edge" cases and specific scenarios that will require approaches that do not align with the guidance presented here. However, we encourage you to make best efforts to align with this guidance as closely as possible in order to gain the benefits we put forward, and to isolate deviation from this guidance within your systems such that these special cases can be migrated appropriately at a later date.

When writing this paper, we have assumed that you want to:

Interoperate broadly between platforms and devices.
Integrate your systems with Microsoft products and technologies.
"Future-proof" your systems so that you can adopt and integrate with tomorrow's technologies without requiring massive investment in time and effort.
Compose your applications into autonomous, cooperative services that can be independently deployed, managed and versioned, enhancing your ability to adapt rapidly to changes in the business' environment.

Guidance Summary

The following sections will detail our guidance, but in summary:

Build Web services using ASP.NET Web services (ASMX).
- Enhance your services with WSE if you want support for advanced Web services.
Keep components within the boundaries of your services.
- While distributed components are viable, assume you will deploy, version and manage systems that share component definitions together.
- Use Enterprise Services where appropriate.
- Use .NET Remoting where appropriate.
When building queuing technology, develop applications against the .NET System.Messaging APIs and/or Enterprise Services Queued Components.
Use BizTalk if you need to orchestrate loosely coupled business processes and/or to integrate several disparate systems.
Use SQL Server Notification Services if you need to provide data-driven notifications from your application.
Use Host Integration Server to access IBM mainframes and MQSeries.

Aligning with this guidance will also greatly simplify your ability to adopt "Indigo", Microsoft's next-generation Web and distributed systems technology. A brief introduction to "Indigo", along with links to other information can be found in A Brief Introduction to "Indigo".

Build Code Using .NET

We strongly advise you to write code by using .NET wherever possible. Applications built with .NET enjoy a number of benefits over unmanaged applications. Many of these benefits are discussed in the Technology Overview.

By following the guidance presented in this paper, you will be better prepared to adopt some of the powerful new technologies that Microsoft is creating to increase your productivity and vastly improve your systems' interoperability.

Build Web Services

We also strongly advise that you build services as Web services using Microsoft ASP.NET Web services (ASMX). Exposing your services as Web services offers a number of benefits, including:

Broad interoperability: Web services can be called from any system that is able to send and receive WS-I Basic Profile 1.x compliant SOAP messages—this includes practically every platform today.
Loose coupling: Web services do not share types. Instead, they expose:
- A contract defined in XML (WSDL) that defines what the service looks like, what operations the service provides, and what information is exchanged during a call.
- XML Schemas for each of the data structures exchanged between service and caller are also provided.
  
  The sharing of schemas and contracts results in Web services being loosely coupled, in that services and callers can be implemented, deployed, and versioned independently using the technologies that make the most sense to each development team. How a service serializes and deserializes messages to and from the wire is an internal implementation detail that external services are not concerned with, just so long as the contracts and schemas are honored.
Future-proofing: "Indigo" services will interoperate with any system that converses using WS-I BP compliant protocols on the wire, including Web services built on ASMX. It will also be very easy to upgrade your ASMX code to "Indigo" when you decide that you need the advanced features that "Indigo" provides you.

Enhance Your Services with WSE If You Need to Support WS-*

One of the limitations of today's "first-generation" Web services is that they don't provide a mechanism for end-to-end data security or integrity. It should be no surprise, therefore, that the first advanced Web services protocol defined was WS-Security.

If your application requires messages to be signed and/or encrypted, then you should enhance your ASMX service using Web Services Enhancements (WSE) for Microsoft .NET. WSE is a .NET technology that enables you to adopt WS-* protocols within your systems today. With a small amount of code and configuration, you can secure your Web service traffic against prying eyes or malicious network agents.

Importantly, WSE 3.0, which will ship shortly after the .NET Framework 2.0, will be wire-level compatible with "Indigo". This means that if you enhance your ASMX services with WSE 3.0, they will integrate with "Indigo" without requiring any code changes!

Keep Objects and Components Inside Your Services

To ensure broad interoperability and integration with existing and future systems it is important to ensure that objects remain inside your services and are not exposed beyond your services' borders.

Object technologies are appropriate for building complex, tightly coupled, high-performance components for use within a service, but as highlighted earlier in this paper, they are not appropriate for use within a widely distributed, or Service-Oriented, system.

Use Enterprise Services & COM+ When Appropriate

Enterprise Services (ES) provides access to the rich services of COM+ for .NET developers. ES is a rich library of utility classes and .NET attributes that enable developers to easily and quickly integrate services such as object pooling, Just-In-Time Activation (JITA) and distributed transactions into their code.

Your investment in developing with ES is protected because you will be able to expose your ES/COM+ components as "Indigo" services by running a simple command-line tool—no code changes required! In order for the command-line tool to build the service stub, you should avoid implementing:

Interfaces that accept object references as parameters
Interfaces that accept types that are not compatible with the .NET Framework COM interop conventions
Interfaces that utilize the "application pooling" feature of COM+ 1.5
Interfaces on components that are marked as "private" to the application
COM+ infrastructure interfaces
Interfaces from the system application
Interfaces from managed components that have not been added to the Global Assembly Cache

Also, if you have unmanaged client applications implemented in, for example Visual Basic 6 or C++, or even script embedded within Office documents, you can modify the code to call "Indigo" services using the "Indigo" COM moniker. With "Indigo" installed on the calling machine, simply change any CoCreateObject(…) [C/C++] or CreateObject(…) [Visual Basic 6/VBScript] statements to CoGetObject(…) [C/C++] or GetObject(…) [Visual Basic/VBScript] and your existing unmanaged applications will make calls to "Indigo" services using the "Indigo" wire protocols.

Flowing Transactions

One of the most commonly misunderstood concepts with ES/COM+ components is around the nature of transactions. A transaction in this context is a two-phase commit transaction often associated with databases such as SQL Server or other transacted resources including MSMQ. The purpose of transactions is to ensure that two or more changes applied across two or more transacted resources are completed atomically, or if there is a failure or problem detected during the transaction, that all changes are reversed so that no partially updated data is left in the system.

Developers can implement ES/COM+ components that initiate a new transaction when called, or, importantly for this conversation, to enlist within a transaction that a caller initiates. In this latter scenario, the caller seamlessly flows the transaction context to the COM+ object during connection or during a method call. This makes writing transacted code very easy compared to building transaction infrastructure from scratch. If you do truly need to ensure the integrity of several updates against several transacted resources during a given operation, then ES/COM+ will make the task a great deal easier.

However, it is often assumed that only COM+ offers transactional features. This is not true! ASMX Services can initiate transactions using just a few lines of code and can then update multiple transacted resources atomically. What you cannot currently do in ASMX is flow transactions created by the caller into a Web service as you can with COM+ components. Flowing transactions into Web services is a feature offered by "Indigo".

Many developers who build COM+ applications implement code that simply calls a stored procedure within the database and returns. The stored procedure does all the transactional work locally. In this case, you should certainly look at implementing such logic within ASMX Services in order to avoid an unnecessary process or machine hop for each call made against your service, and to simplify your code and your systems. The benefits of ASMX noted above will also directly apply to this scenario.

Components that initiate transactions but don't touch multiple transacted resources or perform transactional processing are also prime candidates to be moved to ASMX Services.

By factoring code that truly requires distributed transaction support away from code that does not, more architectural and deployment options will be enabled than will entwining transactional and non-transactional code.

Use .NET Remoting When Appropriate

.NET Remoting is particularly well-suited to situations where you need to perform customized message exchange between your application and another—when implementing a proprietary or custom networking protocol or data encoding mechanism, for example.

.NET Remoting also provides the fastest mechanism by which objects in different app domains within the same Windows process call one another. If you absolutely need to make a large number of calls between components residing in different app domains within the same process in as short a time as possible, .NET Remoting is definitely the best choice. However, it is important to remember that while there is a perception that call-time is very important, if the called service performs significant work, it is likely that the performance differences between using .NET Remoting or ES or even ASMX will tend towards insignificance.

Also, while performance is often cited as a primary reason for choosing .NET Remoting, you should note that while .NET Remoting is extremely fast for making calls across app domains within the same process, ES/COM+ outperforms .NET Remoting for cross-process or cross-machine communications. Also, if you wish to communicate via Web services, note that ASMX outperforms .NET Remoting components hosted in IIS configured to talk SOAP.

.NET Remoting can be used in many other ways. Developers often use .NET Remoting to make calls on objects residing on remote processes or remote machines via TCP or HTTP (or custom transports of their own making) using SOAP or Binary encoding on the wire as per their needs. If you are using .NET Remoting today and have no interest in interoperability or integration with future technologies such as "Indigo", then feel free to continue to enjoy .NET Remoting long into the future.

Remember that .NET Remoting is primarily intended as a .NET Remoting-to-.NET Remoting communications technology and is not appropriate for use where broad interoperability is desired. When .NET Remoting was built, its SOAP encoder was implemented to support RPC-Encoded SOAP. Since then, the industry has chosen to standardize on Document-Literal SOAP encoding (which is incompatible with RPC-encoding); so even though .NET Remoting implements a SOAP encoder, its ability to interoperate across platforms will diminish with time as other platform vendors move towards document-literal encoding. It is important to note that Remoting also does not integrate on the wire with other Microsoft technologies, such as ASMX, WSE, ES/COM+, or MSMQ. Therefore, be sure to isolate your use of .NET Remoting to areas of your systems that you know will never have to talk to anything other than another .NET Remoting endpoint.

.NET Remoting is, by its very nature, a tightly coupled technology (as is ES/COM+ above). Remoting components are not easy to decouple—they even assume that both caller and server are running not only the same version of the .NET Framework but also that they both have a copy of the same version of the DLL implementing the .NET Remoting component installed.

.NET Remoting is a powerful technology that has won many hearts and minds since its introduction in the .NET Framework 1.0. Remoting is interesting if for no other reason than it essentially gives you a glimpse of Microsoft's last "distributed object" technology prior to the introduction of "Indigo".

Therefore, we recommend that you use .NET Remoting wisely, for custom message processing or in-process, cross-application-domain communications. Prefer ASMX instead wherever possible, or ES/COM+ where you need rich distributed-object semantics.

Use System.Messaging/MSMQ for Queued Messaging

Microsoft Message Queue (MSMQ) is a mature and powerful technology that has been included within the Windows Operating System for many years and provides buffered, asynchronous, bi-directional communications infrastructure. System.Messaging (SM) is part of the .NET Framework that enables developers to build applications that benefit from the powerful features in MSMQ.

MSMQ is a valuable part of the Microsoft future distributed technology stack. You should feel secure in the knowledge that the MSMQ applications you build today have a solid foundation long into the future. Microsoft is currently making significant investments in MSMQ to provide a much-improved release targeted at Windows "Longhorn".

Also, your investment in MSMQ will be insured for the future: "Indigo" will ship with support for two MSMQ channels:

MSMQIntegrationChannel: Will enable "Indigo" to integrate into an existing MSMQ application
MSMQTransportChannel: Will provide "Indigo" with a secure, buffered, asynchronous, reliable messaging transport for use in scenarios where reliability, availability and fault tolerance are a major concern

The Future of Microsoft Distributed Systems Technologies

It is important to note that the current Microsoft suite of technologies discussed so far in this paper (ASMX, WSE, ES/COM+, System.Messaging/MSMQ, and .NET Remoting) will remain an important part of the platform for many years to come. None of the technologies discussed so far are going to be removed from the Windows platform any time soon and you can feel secure in the knowledge that many of these technologies are receiving continued investment to ensure their integration with future technologies.

But what about the future? What's coming next?

A Brief Introduction to "Indigo"

"Indigo" is the codename for Microsoft's next-generation distributed systems technology, built on Web services. "Indigo" unifies the most important capabilities of Microsoft core distributed technologies (ASMX, WSE, Enterprise Services/COM+, System.Messaging/MSMQ, and .NET Remoting) into one clean framework and infrastructure, significantly simplifying the process of building distributed, Service-Oriented systems. For an introduction to "Indigo", see the MSDN whitepaper, Introducing "Indigo": An Early Look.

"Indigo" is a very important part of Microsoft distributed systems technologies and following the guidance presented in this paper will greatly ease your adoption of, integration with, and upgrade to this powerful technology.

"Indigo" will be contained within "WinFX", a package containing the .NET Framework 2.0, "Avalon" (Microsoft next-generation UI technology) and "Indigo". WinFX will be released alongside Windows "Longhorn" in 2006, and will also support Windows XP and Windows Server 2003. The WinFX beta program begins in the summer of 2005 and will be preceded by a Community Technology Preview (CTP) release in March 2005.

Much more detail will begin to emerge about "Indigo" from March 2005 onwards and you should regularly visit the Longhorn SDK Web site, as well as the MSDN Web Services Developer Center to keep up to date with the WinFX Beta and CTP releases.

Associated Products

While the above sections discuss Microsoft core distributed systems technologies, several other Microsoft technologies are closely related to this subject. The following sections provide guidance on these other products in relation to the guidance presented in this paper.

BizTalk Server

BizTalk Server (BTS) is the Microsoft business process orchestration and integration engine. BTS offers several powerful features that are useful when integrating a set of disparate components and services that span several platforms and technologies including orchestration, message transformation, and secure, reliable, asynchronous message delivery. BTS is ideally suited to orchestrating complex and rapidly changing business processes through use of its Orchestration designer and powerful orchestration services.

BTS integrates systems via a wide variety of adapters. A BTS adapter for "Indigo" will be released around the same time as "Indigo" is released and will enable BizTalk Server to orchestrate and integrate "Indigo" services into an existing business system.

Future versions of BizTalk Server will build upon "Indigo", integrating it more deeply within the BizTalk Server product, and will reuse "Indigo" as its communications infrastructure, simplifying the task of integrating your business processes further still.

SQL Server

Microsoft SQL Server offers three technologies that are conceptually related to some features offered by Indigo:

SQL Service Broker (SSB)
SQL Notification Services (SNS)
SQL Query Notification (SQN)

SSB is an asynchronous, high-performance, reliable, fault-tolerant, transactional communication between SQL servers and effectively provides a buffered "data queue". SSB generates messages that are primarily optimized for the efficient transfer of SQL data. Indigo services can be built to integrate with SQL and to broker the resultant data back to callers securely, reliably, and over a number of transports and wire-protocols in a highly scalable manner.

SNS is essentially "pub-sub" for the database. SNS enables applications to send queries to the server that will asynchronously send notifications to subscribers when an event that they have subscribed to occurs. These notifications can be presented in a variety of formats and can be delivered via a number of transport means. For example, MSN uses SNS to deliver traffic reports to users via e-mail or text messaging.

SQN is a form of client-initiated change notification whereby queries are sent to the database and the database sends a message back to the caller when the data in the underlying tables change.

In the future, SQL Server's services will integrate with Indigo in order to take advantage of Indigo's powerful messaging capabilities.

Host Integration Server

Microsoft Host Integration Server 2004 helps customers integrate IBM host applications, data sources, messaging, and security systems with new solutions developed using the Microsoft Windows Server System platform.

Host Integration Server 2004 features and technologies, including network integration, host access with enhanced security, and application integration, enable Windows developers to publish business processes in IBM mainframe and AS/400 applications as XML Web services, which brings their Host applications and processes into a Services-Oriented Environment.

Indigo and HIS will, in the future, complement one another and integrate together to enable smooth integration between the Windows platform and host environments.

Summary

In this paper, I outlined the motivations for adopting a Service-Oriented approach to designing and building distributed systems. I discussed how and where to most appropriately use the current Microsoft suite of distributed technologies to implement these systems. My summarized guidance is as follows:

Build services using ASP.NET.
- Enhance your services with WSE if you must support WS-*.
Keep components inside your services.
- Use Enterprise Services where appropriate.
- Use .NET Remoting where appropriate.
Use MSMQ for queued messaging.
Use BizTalk if you need to orchestrate loosely coupled business processes and/or to integrate several disparate systems and technologies.
Use SQL Server Notification Services if you need to build a data-oriented, asynchronous, queued application.

By following this guidance you'll be well aligned with Microsoft strategy. This will enable you to adopt and adapt the current Microsoft technologies and take advantage of the powerful distributed systems technologies we will be delivering in the future to help you build, deploy and operate your distributed services more easily than ever before.