Agreement and Organization: Protocol Architecture for B2B
Microsoft Visual Studio .NET
Microsoft BizTalk Server
Summary: Create a more robust and efficient interaction between a business and its partners by using the architecture of communication protocols to structure business interactions, as well. (13 printed pages)
In this document we argue that today's approach to business-to-business (B2B) communication automation is a major stumbling block to achieving true breakthroughs in e-commerce. First, data alone is not enough to clearly communicate expectations between the participants in an interaction—that is, to specify agreement. Emerging standards have taken a step in the right direction by adding process descriptions to the specification of business services, but, even assuming agreement on a suitable description of a single interaction, the second problem quickly arises: organizing the many interactions between a business and its partners. To solve these problems and improve automation of business interaction, we propose applying the architecture of communication protocols to business interactions. This architecture represents an approach that has been successfully applied in network communication to solve the problems of agreement and organization.
This approach solves both the agreement and organization problems:
- To solve the agreement problem, protocols specify communication rules.
- To solve the organization problems, this layered architecture "divides and conquers" complex interactions into separate but connected conversations.
In the last 10 years, supply-chain initiatives—from lean manufacturing to Six Sigma quality management—sprang up like mushrooms, occupying the hearts and minds of business executives all over the world.
Figure 1. An evolution of supply chain management initiatives
These initiatives reflect common trends spanning industries, company size, and location:
- Market demands are pushing companies towards a collaborative network with each focusing on its core competencies. There are two common approaches to achieve this focus:
- Integrating suppliers as partners into the value chain.
- Outsourcing non-core, yet required, business capabilities (such as payroll) to business process service providers.
- The launching of immense efforts—like the supply chain examples above—to deeply integrate value chain partners and service providers into an interconnected and organized unit. In essence, forming virtual organizations.
Given the resulting density, complexity, and variability of relationships between partners both internal and external, it is clear that simple, linear approaches are insufficient to address those complex, intricately interleaved structures and processes.
For example, Electronic Data Interchange (EDI) and others simply organize and transfer information packaged as business documents. The context of the customer relationship, the associated functions and processes, are not part of this standardization. Initiatives like RosettaNet have improved on that basic approach, defining simple partner interface processes for several business cases, for example. While standard documents and partner interfaces are necessary to solve the interaction conundrum, however, they are not sufficient to completely provide the required solution.
What is there to do? How do we create an environment where we can handle the requirements of individualization, diversity, and complexity, without rapidly growing integration and interaction costs that put further pressure on the bottom line? How do we capture the growing automation, interaction, and communication abilities of all actors in the partner network to bring marketable value?
Web services technology (http://msdn.microsoft.com/webservices/) and service-oriented architecture (SOA, http://msdn.microsoft.com/library/en-us/dnea/html/EaAppConServices.asp) provide a solid technical foundation to develop an environment that supports rich and automated interaction. There are as yet few methods, however, that support the interaction architect and designer in their daunting task of providing solutions to the above problems. We argue that the nature of the problems facing communication in a B2B environment is similar to the issues faced by the designers of network communication systems; therefore, we apply the principles of protocol architecture to business interaction. The cornerstones of the approach are, first, defined agreements for B2B interactions, and then a method to organize a large number of these interactions.
I Say Tomāto, You Say Tomäto
One of the primary issues in virtually all B2B efforts is the inability of the existing approaches to communicate clearly the expectations between participants. The EDI model—with its exclusive data-oriented focus—has been the predominant model for most B2B efforts. More recent initiatives have more or less just replaced one format (EDI) with another, using new Internet technologies. While the move to standard technologies—mostly XML—reduces the costs involved in data processing, it is but a first step.
An informal measure of the complexity of establishing communication between businesses using the existing data-centric format is the user documentation for the Ariba cXML (See cXML 1.2 for example), purchase order (PO), and related documents. The documentation provides the answers to the fundamental questions: "What is the meaning of a PO? For me? For my partner? For other constituents involved in the interchange?" It takes roughly 100 printed pages to describe the interactions related to these business documents that you must understand to use this standard successfully.
This example shows that the apparently simple task of describing a PO is anything but simple. In order to communicate successfully, all involved parties have to understand at least the following elements:
- Purpose of the interaction: Purchase or sell (depending on point of view) the product at a certain price, quality, and date and time.
- Assumptions about the roles and communication channels: Who sells? Who buys? How and when do messages arrive? Is encryption required?
- Vocabulary: What are the messages that make up the conversation? Who sends and receives what message? What language is used? Is the conversation using a particular standard (for example, cXML)?
- Message format: How are the messages encoded? What are the required specific message headers and details?
- Communication rules: When do I expect an answer to my initial request? What is the correct set of answers?
Clearly, the buyer and supplier have to understand and honor these expectations to conclude the interaction successfully. In addition, other parties possibly involved in this interchange—logistics providers, for instance—have to share the same understanding to provide delivery at the right date and time. To make matters worse, not only do all parties involved have to read the documentation in the first place, they have to draw the same conclusions from the documentation, and then apply them to their respective electronic data interchange systems in a compatible manner.
Existing B2B communication systems do not recognize in a systematic way this extended definition of agreement. They limit their scope to the definition of encoding and vocabulary, and rely on ad-hoc agreement, informal documentation, and other means to eventually effect communication between the partners' implementations. This leads to complex, long, and error prone integration projects, as human interpretations form the basis for automation.
A Tangled Web of Interactions
An actual business interaction is made up of many such separate agreements. Figure 2 shows a complaint interaction between supply-chain partners. You can see that just specifying the agreement governing a single interaction (the expanded view in the middle of Figure 2 shows just a small part of such an interaction) is only a small part of the problem. In general, the required, available, or desirable business functions and their associated processes are complex, unclear, and non-transparent.
Moreover, it is important to note that the picture only represents a single customer/supplier integration diagram for complaint management without the required integration into the supplier's (or the customer's) associated business functions and systems. Not only are B2B interactions complex, but they are also very non-standard. These processes reflect the varying and individual relationships between partners (Hammer, M.; Champy, J.: Reengineering the Cooperation; Harper Collins Publishers; New York; 1993; Chapter III), including the push to create value in these relationships through personalized customer service. So, in addition to specifying agreements around single interactions, you must also deal with the complexity of managing myriads of such interactions.
Figure 2. Single supplier/customer cc interaction before project
Obviously, we need some methodology that will let us abstract and manage these interactions. This requires making the individual processes transparent in terms of actions, and not just data. Most current approaches do not do this.
Originally, the network communication community faced similar problems: a rapidly increasing number of endpoints, along with rising functional requirements. The two core problems were interoperability between endpoints, and an overall architectural organization that could make sense of the many types of interactions required to support rich functionality. The answers to these problems are:
- Network protocol models that formalize the agreement required by interoperability.
- Communication service models and layered architecture, to relate the functions of the many network protocols, and produce a whole that is more than the mere sum of its parts.
Over the next sections we apply these ideas to B2B interactions.
Gerard J. Holzmann, in his book Design and Validation of Computer Protocols (1991, Prentice Hall software series, ISBN: 0-13-539925-4), defines protocols as "sets of rules that govern the interaction of concurrent processes in distributed systems."
We will use the simple example of complaint management to demonstrate that B2B interchanges are indeed a problem in protocol design, and that the words business and protocol need to connect in order to move B2B forward. Figure 3 shows a simplified picture of a customer complaint transaction (the expanded view from Figure 2). A protocol governs each interaction—complaint processing, managing quality improvements, crediting the buyer, and so on—required to complete the transaction.
Figure 3. Simple complaint management interaction (from Figure 2)
In order to resolve the issue of a defective delivery, the customer will have to express his complaint to the supplier by providing detailed information about the defective items, associated purchase order, priority of the problem, and other details. The supplier, in turn, will have to respond to this complaint request by either accepting or declining the complaint outright. Alternatively, the supplier can initiate research by requesting additional information before taking any definitive decision on the complaint. Clearly:
- Both entities interact with each other, each time responding to a specific request with answers—potentially consisting of multiple responses like those outlined in Figure 3—that satisfy the demand in question.
- The interaction moves concurrently—each entity making its own decisions, but generally relating to a previously received event—until a final decision point is reached.
- The interaction is distributed.
- The interaction is a process comprised of a series of actions leading towards a particular goal.
Having conclusively established that B2B interactions meet the definition of protocol provided by Holzmann, we need to examine the details of such an approach, and its consequences and benefits for B2B interactions.
Now that we understand the business interaction, let us turn to a more precise inventory of all the elements required to describe the interaction in a way that is amenable to processing by automated systems.
Note Holzmann includes a definition of the service provided by a protocol in his canonical definition. We defer a discussion of this critical aspect to the "Organization" section.
Informally, the interaction illustrated above reads as, "A customer sends a complaint to a business. The business acknowledges the request upon receipt. Depending on the priority of the complaint, the business has from 24 to 72 hours to resolve the complaint."
Assumptions about the Environment
The "environment" for our example consists of two users of the complaint management service, and a transmission channel. The users can be assumed to simply submit a complaint and await its confirmation. The transmission channel is assumed to not lose, duplicate, insert, or reorder messages.
The protocol vocabulary defines the allowable messages in a given exchange. In our example there are six distinct types of messages:
- COMPLAINT for a message with the request for action.
- ACK for a message combined with a positive acknowledgement of the COMPLAINT.
- ACCEPTANCE for a message with the positive or negative confirmation of the COMPLAINT.
- DUNNING from customer if supplier does not reply in allotted time frame.
- INFORMATION REQUEST for a request for more information
- INFORMATION for customer to reply to the INFORMATION REQUEST.
It is important to reach agreement between partners about the encoding standard (EDI, XML) for the agreed upon messages. Note that the vocabulary (UN/EDIFACT, http://www.unece.org/trade/untdid/welcome.htm, cXML, http://www.cxml.org/) of the interaction is quite independent from the encoding. In fact, it is critical to keep both separate, as other circumstances for similar business situations might require different implementations.
Rules authoritatively define principles set forth to guide behavior or action. Even our simple example needs sets of rules that require understanding in a common fashion by both parties to conclude the desired business transaction successfully. Formalizing these rules lets us abstract the interaction, and then automate it.
Events and Sequences
First, you must know about the incoming requests and the possible or correct responses. Then you must define the sequences of events to answer the requesting event with the correct response. It is possible to describe multiple valid answers for any particular event, but the answer has to be part of a pre-defined set of valid answers. Both parties have to share the complete understanding of the supported set of sequences for the communication to succeed.
Events are high-level identifiable occurrences within any given interaction. Usually, each event has data associated with it, carrying the required details for the interaction. Today this data is generally formatted as XML messages. Note that some of those details sometimes impact behavior. For example, in the case of a complaint interaction, the priority field is generally part of the complaint data. This simple numerical field describes an expectation from the customer: "I received defective goods which impeded my production. Please remedy this situation in a timely and hassle-free manner." The supplier will have to understand and honor this expectation in order to keep a successful business relationship.
Simply following the prescribed sequences of events will not provide the parties with enough details to successfully complete business. In most business scenarios—as with our complaint example—the proper response has to happen within a defined interval. If there is no reaction to the initial complaint message in the time interval designated by the agreement, severe financial and business relationship consequences might arise. Note that unlike our simple example, a complete and correct complaint management process has to describe the exception handling process.
Potentially hundreds of events arrive at any given moment for the system to process. The question is, "Which event is associated with which process?" The example above uses a complaint tracking number as the data to find the complaint process that is managing the particular complaint. An analogy is the inclusion of a reference number in normal business correspondence. If one closely reviews the interaction pattern shown in Figure 3, the purpose of this part of the protocol design is to add meaning, or maybe an even better word would be context, to the individual messages, and to provide an end-to-end view of the interaction. This view allows both parties to understand the expectations required for successful interaction, and prepare accordingly.
Correlation focuses exclusively on matching the right request to the correct business process. There is no aggregation at this level.
Procedure rules for the complaint interaction
The procedure rules for this sample interaction are quite simple. Following the established elements for rules above, the protocol reads as follows:
The interaction consists of three required events (E), three optional events (E), and three constraints (C) where "c" is the customer and "s" is the supplier:
- E: (c) sends a COMPLAINT to (s).
- E: (s) acknowledges (ACK) the receipt of the complaint message to (c).
- C: (s) acknowledges the complaint to (c) immediately upon receipt.
- E: (c) can send DUNNING reminder to (s).
- C: (s) does not respond to (c) in required timeframe.
- E: (b) can request INFORMATION from (c).
- E: (c) If INFORMATION requests have been sent, (c) has to respond with requested INFORMATION to (s).
- E: (s) has to CONFIRM (accept or reject the complaint) to (c).
- C: (s) must accept or reject complaint within the priority dependent timeframe.
Now that we have seen how to resolve issues surrounding agreement, the question is how to encapsulate the interaction and create an abstraction, which we can manipulate to build elaborate yet manageable systems handling many such interactions.
Describing the interaction pattern is an important part of the business protocol design, and quite sufficient as a discipline to improve the execution of a single business activity (for instance, "CC processing"). The "CC processing" activity in itself will not successfully resolve the complaint, however. In addition to the initiating the complaint—as described in the "CC processing" activity—the resolution requires quality management improvements (for example, training of supplier employees, repairing defective machines) and the exchange of credit notes. Figure 4 outlines the set of additional business activities required to resolve a single complaint:
Figure 4. Individual services for the complaint management system
In this drawing, the "CC processing" service depends upon further activities—namely quality management and credit management—to achieve closure. It would not make sense, however, to overload the complaint initiation represented by the "CC processing" service with the resolution process, as those processes could differ based upon the complaint, the partner, or geographical proximity. Factoring the processes, as suggested in Figure 4, makes both business and technical sense, as it allows for flexibility and extensibility.
As the figure shows, however, the interaction covering the "CC processing" is isolated, and not related to any of the other services comprising the complaint resolution. How do we bridge the gap from the individual, self-contained interactions, into a full-fledged business interaction that actually allows partners to conduct business?
The key lies in aggregating the segments into a useful whole. Generally this requires identifiers that are shared or related, like a unique complaint number, and collective understanding of the actual status of the business interaction. That is, the complaint has to be acknowledged or rejected before quality management or credit services are even relevant. Before we see how to do this for the complaint management example, we will first describe how the notion of service can be effectively used to encapsulate interaction.
Typically, services provide both the business logic and the state management for a specific problem (For an overview of services visit http://msdn.microsoft.comlibrary/en-us/dnea/html/EaAppConServices.asp). Good services should effectively encapsulate the logic and data associated with real-world processes (agreement), based on intelligent choices about what to include and what to implement as separate services (organization). Basing services on protocols is the sensible way to do this for B2B interactions.
A central notion in network architecture is communication services. These services offer useful functions such as bi-directional, reliable byte-stream delivery to distributed, communicating parties—transmission control protocol (TCP), for example.
Figure 5. Communication services
In good engineering form, communication services hide the often complex inner workings that provide the functions they offer, as objects in object oriented programming (OOP) have methods. You do not need to know how a method works to invoke it.
Take the example of transmission control protocol (TCP) and file transfer protocol (FTP). Their communication service abstraction is the keystone of the overall layered organization of network protocols. This organization lets you create higher level services that would otherwise need to manage an impossibly complex set of interactions. For example, the FTP service is layered on top of the reliable byte-stream delivery service offered by TCP. This allows FTP to engage in a conversation about file transfer without worrying about managing the ordered delivery of data, and all the other mundane details. As Figure 6 shows, this FTP conversation is of course virtual, since the actual communication path passes through TCP.
Figure 6. Layered architecture allows virtual communication at higher levels
We discuss how to apply a similar overall organizational structure to business interactions in the rest of the paper, for our complaint management example. In this case, the overall complaint process between partners is abstracted to a high-level interaction managing the overall complaint lifecycle. This interaction is supported by three services that manage supporting processes: the initial complaint, quality improvements, and credit, as shown in Figure 7.
Figure 7. Layered architecture applied to customer complaint example
Once you identify the business capabilities required by the interactions and their associations within the given organizations, you can structure them and document their hierarchy. This provides the required basis for transparency, so that you can abstract common functions, and isolate common capabilities in interchangeable layers. Figure 8 shows a highly abstracted example of such a structured map for the customer complaint scenario:
Figure 8. A high-level view of the customer complaint scenario
In our case, a structure emerges of a Customer Complaint system, organized as a service that provides resolution of complaints between customer and supplier. The Customer Complaint service is itself a composite entity managing the complaint through the initial processing, quality management, and financial compensation. Each of these functions is in turn provided by a service. The overall effect is to achieve a better factorization of interactions among the system's components. For example, the Quality Management service now encapsulates all of the detailed QM interactions and allows them to vary (for example, you could replace an ISO process with a Six Sigma one) without affecting the rest of the system.
You can also readily recognize in this structure the traditional organization of network communication systems, where very complicated interactions are managed as layered services. Again analogous to TCP providing reliable stream-based communication to HTTP, the QM service provides quality management to the Complaint service without revealing, at that scope, messy and irrelevant details. Note the similarities between Figure 8 and Figure 7.
It is also instructive to contrast this view on the Complaint system with the one in Figure 2, where the system was factored in a way that presented two principal actors (the customer and supplier company) with very complex interactions between them. Here we have introduced an extra actor (namely the Complaint service), with the substantial benefit of encapsulating much of the interaction complexity.
Benefits of a Protocol Approach to B2B Interaction
Advantages that appear obvious after mapping the sample interaction to the protocol definition are:
- Provide a framework for (more) complete and unambiguous description of B2B interaction: Empirical evidence—primarily based upon reviewing existing EDI implementations—shows that interaction based upon data alone is not enough to provide a solid and clear basis to automate business communication. Specifically, such cases do not include exception handling. In contrast, protocol design forces documentation of all elements required to communicate in a reliable fashion. In fact, protocol descriptions provide the basis for systems to automate interaction. Transmission Control Protocol/Internet Protocol (TCP/IP) allows reliable and universally understood binary data transfer between anonymous nodes in a worldwide network—without complex integration projects—just by following a well-defined and agreed upon set of rules. Likewise, business protocols will form the basis for robust B2B communication. Once such a framework exists, we can use it to:
- Design effective interactions that implement the participating partners' business objectives.
- Share the definitions of those interactions with partners.
- Use the definitions for effective integration with other processes.
- Implement those interactions efficiently.
- Determining required capabilities: Following the discipline of protocol design, specifically the explicit documentation of all required elements in machine-consumable artifacts, allows participating systems to determine required capabilities dynamically, and to report missing abilities.
- Machine verifiability of required input/output: Understanding the requirements of the applied protocol, software can determine if the input/output meets the constraints outlined and can issue corrective action.
Web services technologies (such as XML, SOAP, BPEL4WS) and products like Microsoft® Visual Studio® .NET and Microsoft® BizTalk® Server 2004 provide powerful and affordable tools to implement rich interactions between partners. These, combined with service-oriented architecture principles, form the basis for good software design.
As experience in other areas focused on communication shows, however, designing interactions for automation is a complex and error-prone venture. Drawing upon the lessons learned in communication services design, and using the principles of protocol design outlined in this paper, will allow interaction architects to use Web services technologies and SOA even more effectively.
Taken together, agreement based on clear identification and encapsulation of all aspects of the given interaction, organization of these protocols, and proper segmentation through services will:
- Allow faster reaction to changes.
- Limit the scope of changes.
- Allow partners to customize aspects of the interaction.
About the authors
For the past five years Marc Levy has been working on business process automation. As an architect on the BizTalk team, he was one of the principals for the orchestration features. Before BizTalk, Marc was responsible for COM+ security (contributing author to "Designing Secure Web-Based Applications for Microsoft Windows 2000"). Marc's passion with distributed systems is the unifying theme of this work started in the early nineties at the Open Software Foundation (OSF) with the Distributed Computing Environment (DCE).
As a solutions architect, Ulrich Homann is responsible for the design and architecture of Web services and integration of the .NET platform with providers and consumers of business applications. The latter responsibility requires him to engage and interact closely with key partners and strategic customer projects driving key insights into the various affected product groups. It also requires a large travel budget...
Previously, Uli was program manager in the Enterprise Program Management team and responsible for the development relationship with key partners, mainly SAP AG. Uli also served as part of Microsoft Consulting Services, where he was responsible for several key distributed database application projects in Germany.
Prior to joining Microsoft in 1991, Uli worked for several small consulting companies, where he designed and developed distributed systems.
Uli has more than 15 years of experience in the systems industry. He has spent most of his career using well-defined applications and architectures to simplify and streamline the development of business applications. His dedication to this pursuit is driven by a passion to improve product planning by working closely with customers and their applications.