Service Station
An XML Guru's Guide to BizTalk Server 2004, Part I
Aaron Skonnard
Contents
BizTalk State of Affairs
Theory vs. Reality
BizTalk Server Architecture
A Service-Oriented Foundation
Publish-Subscribe Engine
Contract-Driven Messaging
Port Extensibility
Wrap-Up
Who would have believed that XML, such a seemingly trivial technology, could revolutionize an industry? It may have seemed like a long shot in the beginning, but the XML aficionados saw something special and pragmatic right away—a sort of duct tape for the world's information systems. But not all developers felt this way. Some were disappointed by the lack of tool support that would bring XML to life.
Given its place today, however, I don't think anyone could say that XML was all hype. It has revolutionized the way we think about distributed computing, given birth to Web services and service orientation, and continually spawns newer, better tools. One such tool, BizTalk® Server 2004, is poised to finally bring some of XML's benefits to the masses by providing some long-awaited magic.
BizTalk State of Affairs
BizTalk Server 2000 laid the foundation for simplifying common XML use cases in the enterprise. But, needless to say, the final package had some warts. It was hard to install, configure, and operate. Its available tools confused developers and, as a result, was never able to gain significant traction.
Since that humble beginning, the product has evolved significantly. BizTalk Server 2002 came shortly after the initial release, and provided some minor improvements. The big change, however, came with BizTalk Server 2004 when the entire product was rewritten for the Microsoft® .NET Framework. BizTalk Server 2004 came with some major simplifications and a new toolset integrated with Visual Studio® .NET—just the right ingredients for a good developer experience.
And future versions look even better. Microsoft recently released a preview of the upcoming BizTalk Server 2006 (code-named "Pathfinder"), which continues to build on the foundation of .NET, while introducing additional improvements and simplifications. One of the primary goals of the 2006 release is to simplify the user experience, especially the installation, configuration, and operation processes, which have caused so much pain in the past. I've heard some say one of their primary goals was to achieve a SQL Server™-like Next?Next?Next installation. The current release looks very promising.
Theory vs. Reality
One of the first things I noticed after playing with BizTalk Server 2004 is how grounded the product is in reality. Most businesses don't have the luxury of being able to throw away existing investments, so integrating legacy systems with a wide range of communication protocols and data formats is a very big challenge.
While advocates of Web services believe that their approach is the solution to these enterprise challenges, that approach alone would require updating all legacy systems (at least those that don't speak Web services today). If Web services were the answer, everything would have to speak SOAP and the Web services family of specifications. That's a tall order for most enterprises. In addition, most Web services stacks only support a few protocols and are not message-oriented by design, which can limit you when dealing with certain integration scenarios.
The BizTalk folks took a different approach to this problem. Instead of enforcing common protocols, BizTalk Server supports a myriad of protocols through its adapter framework. BizTalk Server provides numerous adapters in the box that address some of the most mainstream use cases, but what's really exciting is the suite of nearly 100 partner adapters you can plug into. You should be able to find one for just about any common enterprise application or technology in use today. Of course, in addition to all of this flexibility, BizTalk Server doesn't require the use of SOAP or WS-*, or even XML. Through BizTalk pipelines and maps, you can support a variety of different data formats to fit your needs, including common flat files and Electronic Data Interchange (EDI).
BizTalk also embraces core XML technologies, such as XPath and XSLT, to deal with tough problems like versioning or alternate data formats to fit a particular customer's needs. In addition, BizTalk provides sophisticated integration with today's evolving Web services technologies when dealing with newer SOAP-savvy applications. Overall, BizTalk Server 2004 offers a message-oriented approach grounded in today's realities.
BizTalk Server Architecture
The BizTalk Server 2004 architecture is neatly organized into two primary buckets: messaging and orchestration. The messaging layer is focused on the details of sending and receiving messages or in other words, "connecting the dots" between applications. In order for an application to participate with BizTalk Server, it needs to be able to communicate via messages. The orchestration layer is focused on coordinating numerous individual message exchanges (see Figure 1), dealing with sequencing and flow, error handling and transactions, and so on. Most BizTalk Server 2004 features and technologies fit nicely into one of these buckets.
Figure 1** Messaging vs. Orchestration **
The messaging architecture provides the foundation for the entire product. You can tap directly into the messaging layer (more on this shortly), or you can interface with it indirectly using the orchestration layer. Orchestrations build on the messaging layer to send and receive the messages used by the business process. Orchestrations are a powerful way to automate complex business processes, but the connectivity and integration benefits come from the underlying messaging components. Nonetheless, there are use cases for BizTalk Server where orchestrations aren't needed, such as in many simple one-way application connectivity scenarios.
It should be no surprise that the messaging architecture is built on an XML foundation. BizTalk Server 2004 can deal with a variety of different message types, but XML is considered the primary or default message type. As a result, BizTalk Server 2004 leverages XML Schema as the primary type system for defining message schemas. And BizTalk Server 2004 provides excellent tool support, integrated with Visual Studio .NET, for working with these XML technologies. XML savvy developers will feel right at home working on this foundation.
A Service-Oriented Foundation
The other thing that really struck me after spending more time with BizTalk Server 2004 was how service-oriented it really is. Most of the folks who promote the tenets of service orientation do so in the context of describing Web services applications and designs. But when you apply the tenets of service orientation to the BizTalk Server architecture, you'll notice an even better fit.
With BizTalk Server, messages are sent and received via ports (see Figure 2). What happens in between is a complete black box to the sender and receiver. Messages are received by "receive ports" and sent via "send ports." Typically, orchestrations are associated with receive ports and are executed when messages arrive. The orchestration may send a result message (or multiple messages) to another system through send ports.
Figure 2** BizTalk Ports and Messages **
Once you have BizTalk Server 2004 installed, you can create and configure ports within the BizTalk Explorer in Visual Studio .NET. Figure 3 shows how to configure a receive port location (which reads from a file directory on disk) while Figure 4 shows how to configure a send port (which transmits to an FTP site). Each one configures the transport (adapter) that will be used along with the address where messages will be sent and/or received.
Figure 3** Configuring a Receive Port Location **
Once the ports are created and configured, consumers who want to integrate with the business process must be informed of the proper message types to send to the receive ports and the message types they will receive back from send ports. With BizTalk Server, this information is shared with consumers using XML Schema, Web Services Description Language (WSDL), and Business Process Execution Language (BPEL) definitions.
Figure 4** Configuring a Send Port **
As you can see, BizTalk Server deals directly with explicit boundaries between applications. In many ways, BizTalk Server is the boundary between the various enterprise applications it orchestrates. Orchestrations also have explicit boundaries (for example, you invoke them by sending messages to ports). The participating applications in a BizTalk environment share schema and contract, not implementation details, to negotiate message exchanges. In fact, BizTalk Server itself knows nothing about how the various participating applications were implemented. Participating applications are completely autonomous and black box to BizTalk Server—they could be running on a variety of different platforms and technologies. New enterprise applications can be added to the mix without affecting existing ones since it's simply a matter of configuring a new message exchange. These traits provide you with a model that can adapt to change over time, which is truly the Zen of service orientation.
Publish-Subscribe Engine
One of the key mechanisms behind the BizTalk Server messaging layer is its flexible publish-subscribe engine. At the heart of BizTalk Server is a SQL Server database, referred to as the MessageBox (see Figure 5). All of the messages that pass through BizTalk Server are temporarily stored in the MessageBox until they are finished being processed. The receive ports have the job of persisting incoming messages to the MessageBox while the send ports have the job of retrieving messages from the MessageBox for transmission.
Figure 5** Publish-Subscribe in Action **
The MessageBox is always in the loop, even for simple request/response style operations. Here's a typical scenario: a message enters a receive port and it's saved to the MessageBox. Then BizTalk Server notices that the message arrived and executes the associated orchestration. The orchestration generates a response message, which is then saved back to the MessageBox. Then a send port picks up the new message and transmits it back to the original sender. So the question is how do send ports and orchestrations know whether to respond to messages arriving in the MessageBox?
This is done through message subscriptions. Both send ports and orchestrations define filters to subscribe to messages entering the MessageBox. When a message hits the MessageBox, all known subscriptions are evaluated to identify matches. The message will then be routed to each orchestration or send port with a matching subscription, as you saw in Figure 5. A single incoming message could be processed by a handful of different orchestrations and send ports. Or you could have a handful of different receive locations (each bound to a different adapter/transport) that receive the same logical message and execute the same orchestration.
Subscriptions are basically filter expressions that return a Boolean result when evaluated. You can use logical ANDs and ORs to group multiple filter expressions together. Figure 6 shows an example of two filter expressions—one that matches the message type (the namespace name plus the local name) and another that checks a value from the message body. Each filter expression is evaluated against the incoming message's context, which can contain a variety of information. It typically contains some BizTalk-specific information (such as port details) along with information propagated from the transport, and any properties promoted from the message body itself.
Figure 6** Defining Filter Expressions (Subscriptions) **
Developers promote nodes in the message schema that they want to make visible to the message context. Figure 7 illustrates how this is done using the Promote Properties dialog accessible from the BizTalk Schema Editor. Once promoted, information about the properties is stored in the schema definition so BizTalk knows what to promote when messages arrive at run time. The promotion process occurs after a message is received by a port but before it's persisted to the MessageBox.
Figure 7** Promoting Properties **
You can think of promoted properties as a view over the entire message as in a SQL view (see Figure 8). This view only needs to contain the information that you need for routing and decision purposes. This makes it possible for BizTalk Server to optimize message processing, which is helpful with large messages.
Figure 8** Message Properties **
This publish-subscribe mechanism makes the messaging layer asynchronous by nature. Synchronous request/response style operations are possible but they're still implemented around the centralized MessageBox in an asynchronous way, which means you'll never achieve the same speeds as traditional synchronous communications. The publish-subscribe mechanism decouples senders from receivers and allows new participants to be added at any point without affecting the existing ones. It also enables some sophisticated content-based routing possibilities through promoted properties.
Contract-Driven Messaging
Now that I've covered the major pieces of the messaging architecture, let's review the high-level development process. Developers typically start by modeling the messages that will flow through their system across the various applications. These messages are defined using XML Schema via the BizTalk Schema Editor. Developers then promote the message properties that will be used for routing or decision purposes. The promotions found in the schema-based contracts are then used by the messaging components to populate the message context at run time and evaluate message subscriptions. Developers can then configure send ports or orchestrations to subscribe to messages matching criteria related to these properties.
As you can see, the entire development process starts with the original schema definitions and everything builds on that. The schema-based contracts dictate how messages are processed, how subscriptions are evaluated, and ultimately how messages flow through the system. They really lay the foundation for the entire solution. It's truly an example of contract-driven messaging at its finest.
Port Extensibility
Throughout this article, I've focused on the BizTalk Server 2004 approach to XML messaging and some of the basic architectural principles that make it all work. I've also highlighted various aspects of the BizTalk Server 2004 architecture that have resonated with me as an XML developer. But this is only the tip of the iceberg.
Figure 9** Port Internals **
In addition to what I've covered here, BizTalk Server 2004 provides some powerful extensibility points that allow you to perform additional processing while a message is being received but before it's persisted to the MessageBox (see Figure 9). It also provides the same extensibility points when messages leave the MessageBox. The three port extensibility points are adapters, where bytes are read; pipelines, where preprocessing is performed in stages; and maps, where the message can be transformed, as shown in Figure 9. XML developers will find these extensibility points particularly interesting due to the flexibility they introduce and the common problems they help solve. I'll cover port internals in detail in the next installment of this column.
Wrap-Up
BizTalk Server has changed dramatically over the years, maturing from its humble beginnings into something more useful and applicable than many developers imagined possible. The current version has many gems for the XML developer better tools and automated support for some of the most common XML integration tasks. BizTalk Server provides a message-oriented approach to connecting systems, something that's truly service-oriented by design. In Part 2 of this series, I'll dive into the port extensibility points including adapters, pipelines, and maps. But while you're waiting, check out "Understanding BizTalk Server 2004", "BizTalk Server 2004: A Messaging Engine Overview", and Pluralsight's BizTalk Wiki.
Send your questions and comments for Aaron to sstation@microsoft.com.
Aaron Skonnard is a co-founder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services, Applied BizTalk Server 2004, and Introducing Indigo courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers.