Guidelines for Application Integration
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.
Summary: This chapter discusses the capabilities of Microsoft products and services that you can use in your application integration environment.
Now that you have examined the requirements for an application integration environment, it is useful to examine some of the products that will help you address your application integration needs. This chapter discusses the Microsoft® technologies that you can use to achieve effective application integration, and how they map to the capabilities described in this guide.
Microsoft provides many products that you can apply to solve application integration problems, including BizTalk® Server, SQL Server, Host Integration Server (HIS), and Microsoft Operations Manager (MOM). In addition, Microsoft delivers numerous technologies as part of the Microsoft Windows Server operating system, including Internet Information Services (IIS), the Active Directory® directory service, Message Queuing (also known as MSMQ), the Microsoft .NET Framework, and various XML technologies.
BizTalk Server 2004 is the main server product used to address application integration. BizTalk Server delivers specific integration technologies, such as mapping and orchestration, and in addition uses many of the underlying services provided by the operating system, SQL Server, and Microsoft Windows® SharePoint Services.
You can also use numerous other Microsoft technologies individually to solve application integration problems, including SQL Server, HIS, MOM, and the many services and features of the Windows Server 2003 operating system itself.
The remainder of this chapter introduces each of these Microsoft technologies and products, shows when you might consider using them, and details the tradeoffs that you should consider when using them.
BizTalk Server 2004
BizTalk Server enables you to define business processes that connect systems, people, and trading partners through manageable business processes. BizTalk Server builds on the Windows Server operating system and the .NET Framework.
You can use the following BizTalk Server features as part of your application integration environment:
- Automation of business processes allows you to reduce human involvement in certain parts of your environment.
- Business Activity Monitoring (BAM) gives information workers a real-time view of running business processes with Microsoft Office tools such as Microsoft Excel.
- Real-time tracking enables you to follow the real-time progress of documents and processes in your BizTalk Server applications.
- Integration with the Microsoft Visual Studio® .NET development system increases developer productivity through a common development environment, and inheriting all the capabilities of Visual Studio .NET and the .NET Framework.
- Integration with the Microsoft Office InfoPath information gathering program enhances information worker productivity by making the most of tools they already know how to use (specifically, applications in the Office suite). InfoPath is a Microsoft Office System program that provides an information workerfriendly front end to BizTalk Server 2004 for entering XML and consuming Web services.
- Single sign-on provides unified authentication between heterogeneous systems and applications (both those based on Windows and those that are not based on Windows).
- Human-based workflow integrates people and processes with a single orchestration engine.
- Support for Business Process Execution Language (BPEL) simplifies cross-platform interoperability for process orchestration with standards developed in conjunction with other industry leaders.
- Support for Web services provides ground-up support for Web services standards such as Web Services Description Language (WSDL) and Universal Description, Discovery, and Integration (UDDI). Referencing and building Web services for orchestration is a simple process in the integrated development environment.
- Business rules enable you to dynamically change business processes to maximize organizational flexibility.
Host Integration Server 2000
Host Integration Server 2000 enables users running Windows operating systems to share resources on mainframes and AS/400 systems without requiring system administrators to install resource-heavy Systems Network Architecture (SNA) protocols on their computers or to install specialized software on the host computers.
HIS provides data integration components, which provide desktop or server-based applications with direct access to host data, including relational and nonrelational mainframe data and AS/400 data, through open database connectivity (ODBC), OLE DB, and Component Object Model (COM) automation controls.
HIS is typically used in application integration to provide components that allow Web-based or Windows-based applications to communicate directly with host-based systems. For example, a BizTalk Serverbased application integration solution can use HIS to Web-enable a host-based application. In addition, HIS also provides a bridging technology between IBM's MQ Series messaging system and Microsoft Message Queuing.
SQL Server 2000
SQL Server often forms an integral part of an application integration environment, providing databases that you need to integrate, but also providing services that you can use directly in application integration. Two of the most important services are Data Transformation Services (DTS) and SQL Server Analysis Services.
Data Transformation Services
Data Transformation Services (DTS) is a group of services delivered by SQL Server 2000 that provide the ability to automate routines that extract, transform, and load data from heterogeneous sources. DTS can extract, transform, and load heterogeneous data using OLE DB, ODBC, or text-only files into any supported OLE DB database or multidimensional store. DTS also automates data transformation by allowing the user to import or transform data automatically on a regularly scheduled basis.
You can access and manipulate DTS operations through interfaces that allow their programs to interact at multiple points during the progression of data transformations. DTS packages can be saved as code in the Visual Basic development system. DTS provides high-throughput parsing and data transformation that is suitable for specific point-to-point integration requirements.
SQL Server 2000 Analysis Services is a middle-tier service for online analytical processing (OLAP) and data mining. The Analysis Services system includes a server that manages multidimensional cubes of data for analysis and provides client access to cube information. Analysis Services organizes data from a data warehouse into cubes with precalculated aggregation data to provide fast answers to complex analytical queries. Analysis Services also enables you to create data-mining models from both multidimensional and relational data sources.
OLAP can be used in combination with the PivotTable Service and Excel (or applications from other vendors) to retrieve and present data about the integration application. BizTalk Server uses Analysis Services in exactly this way, providing both administrative- and business activityfocused reporting.
Windows Server 2003
Windows Server 2003 provides operating system services, Web services, and the.NET Framework programming model. Together, these provide an ideal environment for the development of application integration solutions.
Windows Server 2003 can assume various roles in a network environment, including the File and Print role and the Application Server role. Key application integration services provided by Windows Server 2003 include Active Directory, IIS, and Message Queuing.
In addition to providing directory services to the operating system itself, Active Directory also provides APIs that give access to the data stored in a directory for use by custom applications. For example, you can use Active Directory Service Interfaces (ADSI) to manage application resources from within the directory service. Microsoft products such as BizTalk Server typically use Active Directory for authentication and role-based access control.
With ADSI, you can use high-level development tools such as Visual Basic, Microsoft Visual C#®, or Microsoft Visual C++® to create directory-enabled applications. ADSI can also be used to access other directory services by accessing each one through its own provider. This means ADSI can be used to integrate multiple directories across disparate platforms, such as Lightweight Directory Access Protocol (LDAP), Novell Directory Services (NDS), or the Microsoft Windows NT® version 4.0 Security Account Manager (SAM).
Internet Information Services
The Windows operating system uses IIS to receive HTTP and SOAP requests, and deliver HTTP and SOAP responses to external applications. IIS also serves as a platform for delivering application integration solutions deployed as Web services.
Message Queuing and MSMQT
Microsoft Message Queuing (also known as MSMQ) is a Microsoft implementation of a reliable messaging system. Message Queuing comes with most Microsoft operating systems. Message Queuing provides both a reliable transport (a specific format of message transmission, over a specific TCP port) and physical queues on the server, managed by a queue manager.
Message Queuing supports interoperability with other messaging systems, most notably IBM's MQ Series. These other systems, referred to as foreign messaging systems, can be other Message Queuing systems or non-Message Queuing systems.
BizTalk Server 2004 uses a new technology called BizTalk Message Queuing (MSMQT). MSMQT uses the Message Queuing transport (the same protocol, over the same port), so it interoperates transparently with Message Queuing systems (and thus also with MQ Series systems by using connectivity toolkits).
Microsoft provides full support for XML technologies and has also defined some additional enhancements to XML that can be used with Microsoft technologies.
From a developer's point of view, XML parsers are the fundamental XML component because they act as a bridge between XML documents, seen as a long chain of bytes, and applications that process the XML. Almost all XML applications are built on top of parsers. On Microsoft operating systems, the XML Document Object Model (DOM) is deployed both as a COM component and as managed components.
The parser is responsible for handling XML syntax and, optionally, checking the contents of the document against constraints established in a document type definition (DTD) or schema; the application must understand how to process or display the information. The application is insulated from the details of the XML document, allowing document creators to take advantage of those details without concerning themselves with the application.
By combining XML data with an XSLT transformation style sheet, you can dynamically transform data from one representation to another, and provide a presentation format for the information. XSLT style sheets are similar to cascading style sheets; however, they allow you not only to present XML data, but also to transform it into new data that is tailored specifically to a particular user, media, or client. With XSLT, you can transform data into device-aware or customer-aware information. BizTalk Server uses this technology to convert inbound documents into outbound documents by using XSLT "maps."
A Web service is a programmable entity that gives you a particular element of functionality, such as application logic, and is accessible to any number of potentially disparate systems that use Internet standards such as XML and HTTP. Web services depend heavily on the broad acceptance of XML and other Internet standards to create an infrastructure that supports application interoperability at a level that solves many of the problems that previously hindered such attempts.
A Web service can be used internally by a single application or exposed externally over the Internet for use by a number of applications. Because it is accessible through a standard interface, a Web service allows heterogeneous systems to work together as a single web of computation.
Web services use XML-based messaging as a fundamental means of data communication to help bridge differences between systems that use incongruent component models, operating systems, and programming languages. You can create applications that weave together Web services from a variety of sources in much the same way that you traditionally use components when creating a distributed application.
In the .NET Framework, Web service support is provided by the common language runtime. Many client and server Microsoft applications also either consume or provide Web service interfaces. For example, Microsoft InfoPath 2003 provides a rich-client interface that can use Web services to interoperate with BizTalk Server. BizTalk Server itself is able both to consume existing Web services and to expose business processes as new Web services.
Web Services Enhancements
The basic Web service specification does not resolve many architectural problems such as how to route messages, ensure reliable delivery, or coordinate distributed work using compensating transactions. To address these issues, a collection of Web service architecture protocols have been written by Microsoft, IBM, and others that provide additional functionality. Web Services Enhancements (WSE) is one implementation of (or a subset of) these protocols.
Web Services Enhancements version 1.0 for Microsoft .NET is a new library for building Web services using the latest Web services protocols. This version focuses on the basic message-level protocols: WS-Security, WS-Routing (and WS-Referral), and Direct Internet Message Encapsulation (DIME) and WS-Attachments. WSE integrates with ASP.NET Web services, offering a simple way to extend their functionality. BizTalk Server can use these enhancements by adding custom components in the BizTalk Server pipelines to implement WSE security, for example.
Windows SharePoint Services
Windows SharePoint Services enables you to create Web sites for information sharing and document collaboration. SharePoint sites are made up of Web Parts and ASP.NETbased components. Web Parts are designed to be added to pages and configured by site administrators and users, creating complete page-based applications.
Windows SharePoint Services sites extend file storage, providing communities for team collaboration and making it easy for users to work together on documents, tasks, contacts, events, and other information. Microsoft BizTalk Server uses this infrastructure to enable you to publish the results of Business Activity Monitoring on a Windows SharePoint Services site.
Microsoft Operations Manager
Microsoft Operations Manager 2000 (MOM) is a network monitoring solution that captures and reports events throughout your network. By defining rules, you can automate responses or assign problems to a specific staff member for resolution. MOM displays information about service-level exceptions, open alerts, and computers in the configuration. You can check specific alert and event details, performance data, and monitored computer status.
MOM provides proactive real-time system monitoring for servers and computers throughout the enterprise that are running Windows 2000 and Windows 2003. MOM also displays useful information about computers and their alerts. You can display a view of all computers to see computer properties, alerts, events, performance, computer group membership, and the processing rules applied to each computer in the configuration group.
In addition to monitoring specific aspects of Windows Server operating systems, MOM can also load management packs, which allow it to be extended to monitor various Microsoft server applications.
To determine which Microsoft technologies you should use in your application integration environment, it is important to understand how the available technologies map to the capabilities defined in this guide. The capabilities are defined as follows:
- Business process integration capabilities
- Data integration capabilities
- Communications capabilities
- Security capabilities
- Operational management capabilities
Business Process Integration Capabilities Provided by Microsoft Technologies
This guide defines a number of capabilities that are required for effective business process integration. Table 4.1 shows how Microsoft technologies provide these business process integration capabilities.
Table 4.1: Business Process Integration Capabilities Provided by Microsoft Technologies
|Rules Processing||Custom .NET Framework components, SQL Server stored procedures, and BizTalk Server Business Rules Engine|
|Transaction Management and Compensation||Microsoft Distributed Transaction Coordinator (DTC) and BizTalk Server long-running transactions|
|Workflow||BizTalk Server Human Workflow Services|
|Orchestration||BizTalk Orchestration service|
|State Management||Common language runtime, IIS session management, and BizTalk Server orchestration state management|
|Event Processing||SQL Server triggers, MSMQ Events, and BizTalk Server messaging subscriptions|
|Schedule||Windows Server Schedule service and SQL Server Job Scheduler|
|Contract Management||BizTalk Orchestration service (BPEL4WS)|
It is useful to look at these capabilities in more detail.
When you need to implement new business rules as part of your application integration solution, the business rules vary widely in complexity and in how they are implemented. In typical business applications, you can develop business rules using SQL Server stored procedures or .NET Framework components. In BizTalk Serverbased solutions, low-level business rules are often implemented in the messaging pipelines (validation rules, generally), in the data manipulation functions (data conversion rules, generally), or in the orchestration system (business rules implemented as .NET Framework components, for example).
All of these mechanisms have a significant disadvantage: Changes to the business rules require a recompile and redeployment of the appropriate component(s) that implement the business rule. Another disadvantage of these methods of implementing business rules is that the rules are expressed in terms of implementation artifacts (such as a specific field in an XML schema), rather than in a vocabulary based on the business process itself.
To assist with these issues, BizTalk Server provides a Business Rules Engine through which you design, build, compose, and manage your unique business rules. Business rules are statements that govern the conduct of business processes. The Business Rules Framework enables you to independently and seamlessly change code, processes, and rules by updating rules and the policies that contain them without having to update code as you would in a traditional development cycle.
Transaction Management and Compensation
XA-compliant transactions or Atomicity, Consistency, Isolation, Durability (ACID) transactions are implemented using a two-phase commit protocol and are managed in a Microsoft environment by the DTC. BizTalk Server supports XA-compliant transactions through the DTC. The DTC is used when performing synchronous database updates or when accessing other transactional resources. It also provides support for long-running transactions and timed transactions.
All of the transaction types supported by BizTalk Server provide the ability to specify compensation logic, which allows the results of transactions to be undone or provides an alternate series of actions to take if a transaction is aborted. Compensation logic is necessary even for short-lived transactions, because some of the actions performed within the context of the short-lived transaction may not themselves be transactional. By using compensation logic for failure processing, you can add additional error handling semantics to schedules for all transaction types.
Human Workflow Services
Human Workflow Services (HWS) uses the BizTalk Server orchestration engine and interfaces with Microsoft Office products. HWS provides a workflow model that involves human interaction rather than fully automated business processes. It allows business activities to be captured as a workflow, and is designed to be modular enough to be used as a stand-alone component or as part of a larger business process. Workflows in HWS are created by composing a series of tasks, or actions, such as Delegate, Approve, Review, and Escalate, which are associated with users based on their roles.
BizTalk Orchestration provides a rich programming environment, which includes a visual design and development environment that you can use to separate the business process being developed from the implementation of that process. The environment supports a complete set of programming constructs, including transactions and exception processing semantics, support for long-running processes, and state management.
A BizTalk orchestration is an application running within the .NET Framework managed environment. The application itself uses a graphical programming language with its own specific semantics, which is then compiled to Microsoft intermediate language (MSIL). The graphical programming language has many of the same constructs and rules as a C# application, along with custom constructs designed to suit long-running business applications.
You can specify an implementation for each of the individual actions that make up a business process and compile this information into an executable representation. These business processes are physical processes, which are initiated by some event (such as the arrival of a message) and then may run for an extended duration (weeks or months), sending messages to, and receiving messages from, other systems in the enterprise, or from parties external to the enterprise.
Microsoft products handle state in a variety of ways. Of particular note in application integration solutions is the way in which state is handled in a Web-based integration solution (in ASP.NET), and how state is handled in a .NET Framework application.
ASP.NET provides the following application-state support:
- A state facility that is compatible with earlier versions of ASP.
- An application-state dictionary that is available to all request handlers invoked within an application.
- A synchronization mechanism that enables you to coordinate concurrent access to variables stored in the application state.
- Application-state values that are accessible only from code running within the context of the originating application.
Because a BizTalk orchestration is actually a .NET Framework application, state is automatically managed within each instance of an orchestration, meaning that state information is maintained both within the messages flowing through the orchestration and in local or global variables declared within the orchestration.
BizTalk Orchestration also provides the ability to back up the entire state of an orchestration instance to the database and, when necessary, to restore the orchestration instance with all state information. This mechanism is referred to as dehydration/rehydration and is required to support the deployment of huge numbers of orchestration instances running over an extended period of time.
The message transports available in Windows all provide some form of event trigger on the arrival of a message, which allows applications to react to this event. Message Queuing provides triggers, which bind Message Queuing message arrival to a COM object or to the instantiation of a process. Similarly, SQL Server provides data triggers, which are fired as the result of some data update.
Within a BizTalk orchestration, the arrival of a new message also acts as an application event. A new message arriving at the system will either instantiate a new instance of the business process (instantiate a new orchestration instance), or the message will correlate with an existing instance of a business process. For example, a purchase order acknowledgment will correlate with the instance of the business process to which it corresponds. An event may also be initiated by a .NET Framework component. This mechanism may be used to synchronize the orchestration with external events.
Microsoft products provide various scheduling services. The Schedule service in Windows Server handles application-level scheduling. In SQL Server, SQL jobs are scheduled with a mechanism that uses the SQL Server Agent.
Timed transactions in BizTalk Server provide a specific form of scheduling function, whereby an orchestration that is currently suspended while awaiting the arrival of a message will time out after a scheduled period of time.
When an orchestration enters such a wait period, the orchestration is usually dehydrated to the database and then is rehydrated later when the message arrives, or according to the scheduling after a message timeout. In the event of a timeout, the orchestration performs the appropriate exception handling for the timed transaction.
BizTalk Server provides two key pieces of functionality to support contract management:
- BizTalk Server reliable messaging
- Support for Business Process Execution Language for Web Services (BPEL4WS)
With BizTalk Server reliable messaging, messages that you send to a trading partner are marked up to require a response within a particular time frame.
BizTalk Server can automatically manage both ends of the communication. When BizTalk Server is the originator of the message, it waits for a response from the party to which the message was sent. If there is no message response, it resends the message a configurable number of times. If there is still no response, BizTalk Server marks the message transmission as failed and performs appropriate exception handling.
When BizTalk Server is the receiver of the message, it automatically extracts the return address from the message, formulates a response message, and sends it to the return address specified.
BizTalk Server 2004 also provides full support for BPEL4WS. BizTalk Orchestration provides a superset of BPEL4WS capabilities. In fact, BizTalk Server 2004 can import a BPEL4WS specification from a compliant orchestration and automatically build a compliant orchestration from the specification.
Data Integration Capabilities
This guide defines a number of capabilities that are required for effective data integration. Table 4.2 shows how Microsoft technologies provide these data integration capabilities.
Table 4.2: Data Integration Capabilities Provided by Microsoft Technologies
|Data Validation||XML schemas and BizTalk Server parsers|
|Data Access||ADO.NET and SQLXML|
|Schema Definition||BizTalk Editor|
|Data Transformation||SQL Server DTS, XSLT, and BizTalk Mapper|
|Schema Recognition||XML schemas and BizTalk Server parsers|
It is useful to examine these capabilities in more detail.
Virtually all Microsoft applications support XML technologies centered on the Microsoft implementation of the XML DOM. The DOM defines a standard set of commands that parsers expose so you can access XML document content from your programs. An XML parser that supports the DOM takes the data in an XML document, parses and validates it, and then converts it to a data structure in memory. The data is exposed by way of a set of objects that you can program against. Many Microsoft client applications and server products use the Microsoft XML DOM.
In BizTalk Server, a parser in the receive pipeline performs data validation. The parser component decompiles the message using the information in a supplied schema definition, which is an XSL schema. BizTalk Server provides parsers for XML, custom flat files, and for EDI documents. Each of these parsers uses the XSL schema definition to validate the corresponding document type and convert it to an XML representation.
The .NET Framework includes a new data-access layer that two main groups of classes—the ADO.NET and XML core classes—provide. ADO.NET provides classes that implement database-independent data containers, along with database-oriented tools such as SQL Server and OLE DB commands, managed data providers and connections, data readers, and data adapters. ADO.NET represents the .NET Framework way of working with data, irrespective of data source, data format, or physical location. It also features a data-centric design center that is different from the database-centric vision of ADO and better suited to today's Web programming needs.
SQL Server 2000 was the first version of SQL Server to provide native XML support, which was limited to the more basic XML feature set (template queries, mapping schemas, and OPENXML). The SQLXML add-on provides additional features such as updategrams. SQLXML helps you to build data-driven Web services.
You can use the BizTalk Editor (hosted in the Visual Studio integrated development environment [IDE]) to create BizTalk Server schemas. The schema definition specifies the document structure, individual records, and fields that make up the document. For example, the flat-file parser uses the XSL document schema to determine the type of flat-file document (positional, delimited, or some combination of these) and other document features, such as which delimiters and escape characters the document uses. For each field in the document, the XSL schema definition specifies the type (date, integer, string, or other type), and other validation information, such as field lengths.
Figure 4.1. BizTalk Editor
BizTalk Server provides additional facilities within the IDE that allow you to test schemas against sample instances of those schemas, and to generate sample instances from a schema definition. You can also automatically generate schemas from XML instances, or from XDR schema specifications.
SQL Sever DTS provides a set of graphical tools and programmable objects to help you solve data transformation and data movement problems, including how to extract, transform, and consolidate data from disparate sources to single or multiple destinations. Sets of tasks, workflow operations, and constraints can be collected as DTS packages that can be scheduled to run periodically or when certain events occur.
BizTalk Server provides a data transformation mechanism based on XSLT transformations. These transformations are specified as XSLT maps, which convert a source format to a destination format. You can apply maps in either the receive pipeline or the send pipeline, or within an orchestration itself. XSLT maps are developed using the BizTalk Mapper, which also integrates with Visual Studio .NET. Figure 4.2 shows an XSLT map in the BizTalk Mapper interface.
Figure 4.2. XSLT map in BizTalk Server
The BizTalk Mapper interface enables you to map source fields to destination fields directly by dragging the source field to the destination field. A large number of data manipulation functions (functoids) can also be applied to the mapping.
Functoids are available from the Visual Studio .NET Toolbox, grouped by type. BizTalk Server provides over 75 functoids, which provide a wide variety of functionality, including the ability to map source fields to different format destination fields, aggregate data (for example, total all the line items in a purchase order), and perform various data lookups (for example, look up a supplier name in a database by supplier ID).
This guide defines a number of capabilities that are required for effective communication integration. Table 4.3 shows how Microsoft technologies provide these communication integration capabilities.
Table 4.3: Communication Capabilities Provided by Microsoft Technologies
|Message Routing||BizTalk Server MessageBox and subscriptions|
|Message Delivery||BizTalk Server message ports|
|Message Queuing||Microsoft Message Queuing and BizTalk Message Queuing (MSMQT)|
|Message Forwarding||BizTalk Server MessageBox and subscriptions|
|Message Correlation||BizTalk Orchestration correlation services|
|Message Instrumentation||BizTalk Orchestration|
|Addressing||BizTalk Server message ports|
|Transactional Delivery||Microsoft Message Queuing and BizTalk Server reliable messaging|
|File Transfer||BizTalk Server message ports|
|Serialization/Deserialization||BizTalk Server receive and send pipelines|
|Batching/Unbatching||BizTalk Server messaging pipelines|
|Encode/Decode||BizTalk Server messaging pipelines|
|Connection Management||BizTalk message ports|
It is useful to examine these capabilities in more detail.
Message Routing and Message Forwarding
BizTalk Server 2004 provides a publish-subscribe mechanism centered on a feature called the MessageBox. At the time of delivery, a publisher delivering a message to the system for delivery has no knowledge of the message delivery mechanism, or of where (to which receivers) the message will be delivered.
Messages that the system receives are first passed through a receive pipeline. This pipeline preprocesses the message by decoding, decrypting, and so on. In addition, the pipeline extracts additional data about the message and about the message receiving process. For example, it extracts information about what port the message was received on, details of the transport used (for example, what the file name was for a file-based transport), and authentication information (for example, the user ID if available). At the end of this process, the message, together with all the metadata collected about the message, is placed into the MessageBox.
This architecture allows you to implement sophisticated message-addressing and message-routing designs. To route the message to its destination, the MessageBox processing evaluates the subscription rules associated with each outbound port or orchestration port. For each port that has a subscription that matches the metadata, the MessageBox passes a copy of the message to that send or orchestration port. For example, a subscription may be set up that matches all messages received by the ReceivePO receive port.
In addition to routing messages to specific orchestration or messaging ports, you can use the metadata associated with each message to specify addressing information for the eventual destination of the document. Lastly, you can derive the metadata associated with a message from the content of the message, which makes content-based routing very easy to implement.
Figure 4.3 (following) shows a high-level overview of message flow within BizTalk Server 2004. Message flow through the system starts at a receive location. BizTalk Server makes a distinction between a receive port (which has a logical representation in the context of the BizTalk Server application being developed), and a receive location (which has a physical representation in the context of the deployed BizTalk Server application).
A receive location specifies a particular transport type (or adapter), various parameters specific to the transport type (for example, the file directory and filename mask), and the pipeline that will be used to process the submitted document. The receive port specifies the authentication and tracking for the receive port, along with any mapping you want to apply to the inbound document. There is a one-to-many relationship between receive ports and receive locations, which means that one receives port can have many different receive locations, spanning multiple different transport types.
Figure 4.3. Message flow in BizTalk Server 2004
When a document is submitted to a receive location using an appropriate transport, the message is processed by the pipeline specified for that receive location. A pipeline is a series of logical stages, each of which may be represented by one or more components. The stages in a pipeline represent the processing that the pipeline performs on the document. BizTalk Server comes with several standard receive and send pipelines, with stages to:
- Resolve party
You can customize the BizTalk Server generic receive and send pipelines by adding stages. You can also populate each of these stages with one or more custom pipeline components to perform the activities of that stage. For example, to convert a mainframe message encoded with EBCDIC to ASCII, you can write a custom decoding component.
Figure 4.4. Customizing a BizTalk Server pipeline
After a message is delivered to the MessageBox, it is delivered to each its subscribers. A subscriber to a message is either an orchestration or a send port. An orchestration or a send port sets up a subscription to a message by setting up a message filter, referencing the metadata associated with the message. For example, a send port can set up a filter to receive all messages that were received on a specific receive port.
In addition to send ports, BizTalk Server also uses send port groups, which are collections of send ports. In a BizTalk Server system, each send port can have a subscription, and each port can belong to a send port group. The send port group also has a subscription. When a message is processed, a copy of the message is sent to each subscription in the system that matches the message properties. If the subscription for the send port group matches, the message is sent to each port in the group. If both the send port group and the send port subscriptions match the message properties, the message may be sent to a port twice.
BizTalk Server 2004 uses BizTalk Message Queuing (MSMQT) for message queuing, which interoperates transparently with Message Queuing and MQ Series systems. However, the messages are placed directly into the BizTalk Server MessageBox (just as with all other transports) rather than a queue, and BizTalk Server, rather than a queue manager, manages the transport.
Previous versions of BizTalk Server provided a mechanism whereby an incoming message was associated with a unique moniker that identified that instance of the business process. This mechanism required you either to place a copy of the unique moniker in the message (which was not always possible) or to write additional code to associate some unique identifier in the message (such as the purchase order number) with the unique moniker of the business process instance.
BizTalk Server 2004 uses a mechanism similar to the latter, whereby you designate some combination of fields in the incoming message as unique (for example, you may designate that the combination of the purchase order number and the purchase order date be guaranteed as unique), and this automatically correlates the message with the correct instance of the business process.
BizTalk Server delivers a copy of a message to any ports where the ports' subscription information matches the message metadata. This mechanism provides a dynamic routing algorithm, which is evaluated at run time, allowing you to develop very sophisticated message routing. In addition, orchestrations that specify ports either can specify all transport information for those ports at development time, or can defer this task until deployment, or at run time.
These send ports or orchestration ports specify the transport involved (which may be a true transport, such as HTTP, or may be an adapter, such as the SQL Server adapter), along with the details of that transport (such as the address). For example, a port may be an HTTP port that sends documents to a specific HTTP address. These addresses can be absolute, or they can be specified using keywords, which are replaced at run time with information from the message metadata.
As well as providing a sophisticated queuing and message routing infrastructure, Message Queuing also provides transactional message queue support. Hence, Message Queuing provides a resource dispenser, which operates together with the DTC to provide transactional message queues. These queues can be used to implement transactional delivery of messages.
BizTalk Server also provides transactional messages at various levels of integration. Message sent by BizTalk Server are automatically retried if a transport failure (such as an HTTP error response) is returned. When working with Message Queuing, this provides a transactional activity that spans the entire transport process.
For transports that are not transactional (most are not), BizTalk Server also provides a reliable messaging mechanism, where the receiver of a reliable message must reply to that message within a certain time frame, or else the message will be resent. BizTalk Server handles the message retry and receipt automatically.
BizTalk Server ports manage all transport of messages by using Windows Server messaging services, such as IIS, and the underlying network layer. BizTalk Server supports File, FTP, HTTP, MSMQT, SMTP, SOAP, and SQL Server transports. BizTalk Server does not differentiate between transports such as HTTP and application adapters such as SQL Server 2000. Instead, it handles both types of transport generically as adapters. You can also develop custom adapters to support your file transfer needs.
BizTalk Server uses message processing pipelines to process messages as they are sent and received. A send/receive pipeline commonly has a stage for serializing or deserializing the messages passing through the pipeline. This stage converts the native document from or to XML using a serializer or parser component. BizTalk Server supports XML, custom flat files (both delimited and positional), and EDI document (EDIFACT and X12) parsers. You can build custom serializers or parsers for customized formats, such as HL7 for healthcare applications.
Although the underlying communications model is publish-subscribe, you can still accommodate request/reply style models with BizTalk Server. Request/reply is implemented as a request-response port. This implementation is a variation of the standard one-way receive port, which returns a response using the same transport as the received message. Figure 4.5 shows a simple orchestration connected to a request-response port. The orchestration receives a message (through an HTTP Post), performs some processing, and then creates and sends the HTTP response message.
Figure 4.5. Simple BizTalk Server orchestration connected to a request-response port
You can use similar request-response communication models to implement Web services. In this case, you use orchestration as the implementation mechanism for the Web service.
Document batching and unbatching are supported by BizTalk Server, by messaging pipelines, and through use of XML envelopes. A batched message received by BizTalk Server is automatically broken up into its component messages using an envelope that essentially specifies how to unbatch the document. Individual messages are then processed separately by BizTalk Server, and can then be batched up again using a custom pipeline component
BizTalk Server supports document encoding and decoding through the receive and send pipelines. You can decode documents received by BizTalk Server that have been encoded in a specific format (for example EBCDIC from a mainframe, or SMIME for security) within the receive pipeline, prior to processing by BizTalk Server. Similarly, you can encode documents sent from BizTalk Server using an encoding stage in the send pipeline. You can also create custom encoder/decoder pipeline components for use within the pipelines, such as a Pretty Good Privacy (PGP) component.
BizTalk Server supports a concept of parties taking part in a business process. For example, your organization may do business with numerous suppliers, all of whom receive purchase orders from your organization. You may find that it becomes too complex to manage and administer send ports for each of these suppliers. BizTalk Server solves this problem by implementing the integration application ports in terms of service links.
To add a new trading partner to the application integration solution, you define a new party in terms of the service links required by the application. You specify physical port information for the party, which links to the logical service links.
Microsoft supports bridging between industry standard protocols (for example, HTTP) and protocols from other vendors by using custom protocol adapters. For example, Microsoft has developed an MQ Series adapter that provides bridging support between Microsoft Message Queuing and IBM MQ Series.
This guide defines a number of capabilities that may be required to secure your application integration environment. Table 4.4 shows how Microsoft technologies provide these security capabilities.
Table 4.4: Security Capabilities Provided by Microsoft Technologies
|Authentication||Active Directory and SharePoint Portal Server Enterprise single sign-on|
|Information Protection||Windows Server|
|Identity Management||Enterprise single sign-on|
|Profile Management||Enterprise single sign-on|
|Security Context Management||Active Directory and Enterprise single sign-on|
It is useful to examine these capabilities in more detail.
All resources managed by a Windows Server operating system automatically have authorization checks performed against them whenever a user attempts to access them. The user identity is checked against the access control list of the resource being accessed to determine if the user has the rights to access the resource in that manner. BizTalk Server and all other Microsoft server products use this authorization mechanism.
In addition, you can use Authorization Manager in Windows Server 2003 to facilitate role-based authorization for your applications. Authorization Manager allows you to authorize applications not only against Active Directory, but also against an XML file store.
Users accessing a Windows Server system are required to authenticate. This authentication occurs either as a result of logging on to the network, or for access through a Web-based portal, by authenticating against that portal using standard Web-based authentication methods. Users' security credentials are validated against the credentials maintained for that user in Active Directory.
BizTalk Server manages authentication of users through existing Windows authentication mechanisms. Access to the application integration system itself (for administration, configuration, or for business activity monitoring) requires the user to authenticate himself or herself so that authorization checks can be performed.
For applications or external users submitting documents to BizTalk Server, the kind of authentication available depends on the transport adapter used to receive the message. BizTalk Server supports all common authentication standards for each transport type. For example, when accessing a Web site, authentication occurs through basic authentication, NTLM authentication, certificate-based authentication, or another authentication mechanism.
Decryption and encryption of messages is managed within the receive and send pipelines by specific encryption pipeline components. These components use the encryption technologies that Windows Server supplies by storing and retrieving digital certificates in the server certificate store, and using these certificates to decrypt or encrypt the documents. You can also write custom pipeline components such as a PGP encryption component.
When building complex business processes that interact with many systems within the enterprise, you need a mechanism that enables a user to access multiple heterogeneous applications through one enterprise logon. BizTalk Server uses the Enterprise single sign-on (SSO) provided by SharePoint Portal Server. When using SSO, after a user is authenticated by Windows, no additional credentials are required to access resources from non-Windows systems or other back-end systems and applications.
The SSO system is a distributed environment that consists of a centralized credential database and at least one server. The credential database is a SQL Server database that relates the Windows user to the user's corresponding non-Windows credentials. You can also use the SSO system to store additional information, such as receive locations and send ports configuration information that is specific to a user.
In the same way that business documents can be decrypted and encrypted in the receive and send pipelines, you can use the pipelines to check a received document for a valid digital signature, or to add a digital signature to a document that will be sent to a third-party. Again, this is done by pipeline components, which use Windows Server technologies. Nonrepudiation can be supported by validating the digital signatures fixed to a message and by maintaining the details of the digital signature throughout processing as custom metadata associated with the message.
BizTalk Server 2004 provides the BizTalk Explorer to streamline partner configuration. You use other BizTalk Server tools, such as Orchestration Designer and Pipeline Designer, to create a single common integration application, and then use BizTalk Explorer to vary the configuration as required for different partners. Decoupling the partner relationship from the business logic in this manner increases the reusability of the solution. You can create an orchestration without knowing the specific send ports or receive locations, which can be configured later. Not only can you reuse orchestrations, but you also have greater flexibility to make changes to the partner relationships without having to change the actual applications.
Security Context Management
Windows Server requires users to authenticate and performs authorization checks for all resource access. Windows also allows one security context to impersonate another, given appropriate authentication information. Your integration applications can use this feature to access resources in different security contexts.
Similarly, you can use SSO to supply security context information for systems that are not based on Windows. Here, a security principal provides a security context to access the integration application, and then the application obtains another security context from the SSO system.
Operational Management Capabilities
This guide defines a number of capabilities that may be required for effective operations of your application integration environment. Table 4.5 shows how Microsoft technologies provide these operational management capabilities.
Table 4.5: Operational Management Capabilities Provided by Microsoft Technologies
|Business Activity Management||BizTalk Server Business Activity Monitoring (BAM)|
|Event Handling||Windows Management Instrumentation (WMI) events, Event Viewer, Performance Monitor, and MOM|
|Configuration Management||System Management Server|
|Change Management||BizTalk Server Business Rules Engine and schema versioning|
|System Monitoring||Windows Server and MOM|
It is useful to look at each of these in turn.
Business Activity Management
SQL Server 2000 Analysis Services is a middle-tier service for OLAP and data mining. Analysis Services includes a server that manages multidimensional cubes of data for analysis and provides client access to cube information. Analysis Services organizes data from a data warehouse into cubes with precalculated aggregation data to provide fast answers to complex analytical queries.
BizTalk Server 2004 uses this technology to track and analyze all documents that are transmitted through a BizTalk Server system. In addition, Analysis Services are used to provide business-level reporting to end users.
Business Activity Monitoring (BAM) in BizTalk Server provides business analysts direct visibility into running business processes. BAM does this by concentrating and analyzing data for heterogeneous information sources, and presenting a real-time view of business state, trends, and critical conditions, enabling them to make better business decisions based on more relevant data.
All Microsoft server products provide Windows Management Instrumentation (WMI) events, which can be accessed programmatically, or by a variety of client applications. Your business applications can incorporate their own event handling logic (combining this with application specific event handling logic), or you can use one of the many client applications for event handling.
Windows Server includes Event Viewer and Performance Monitor, both of which you can use to view event data for a business application. For more functionality, you can use MOM, which allows you to automatically monitor a wide range of system and application events, and then perform various actions based on the level and severity of the events captured. You can use MOM to monitor events across a large number of servers in an enterprise.
Systems Management Server (SMS) 2003 helps you to perform change and configuration management for the Microsoft platform, and gives you the ability to provide relevant software and updates to users. SMS 2003 monitors and manages the installed application base, and allows you to push key updates to servers throughout the enterprise. SMS uses the management capabilities built into the Windows operating system.
In addition, BizTalk Server provides you with a full WMI interface to all BizTalk Server artifacts. The definition for these configuration artifacts can be exported as an XML binding file. You can also recreate the configuration at any time by importing this binding file.
Active Directory is the directory service on Windows Server systems. In addition to providing directory services to the operating system itself, Active Directory can be used to provide information to, and store information from custom applications.
BizTalk Server 2004 solutions are deployed as .NET assemblies, which include versioning information. Hence, schemas and maps are all versioned within the assembly that they reside, so when you need to make changes to the system, you manage backward compatibility by referencing specific versions of these components.
Similarly, vocabularies and business rules as deployed by the Business Rules Engine are all fully versioned, so once a business rule has been deployed, it cannot be changed. Instead, a new version of the rule must be created and deployed.
BizTalk Server provides monitoring in two separate ways: by writing WMI events and by writing details of all documents processed into the SQL Server database. You can monitor WMI events using various tools supplied with the Windows operating system, such as Performance Monitor and the Event Log, and by using MOM.
MOM provides management of events and performance of Microsoft servers from a Web console. You can create sophisticated rules to respond to events, generate custom reports, or handle operational tasks using one of the add-on management packs. The management packs provide additional information to allow MOM to manage Microsoft servers. For example, BizTalk Server ships with a MOM management pack that defines the key events, and reports required to manage a BizTalk Server system.
You can use the Health and Activity Tracker (HAT) that ships with BizTalk Server to examine all the details of all documents processed, which are captured in the database. This tool essentially provides a (large) number of stored procedures to query specific aspects of the BizTalk Server database. You use the tool to view all messages being processed or that have been processed, and any running orchestrations. Linked to the HAT is an orchestration debugger, which can be configured to stop at any step in an orchestration, and then to step through the orchestration.
After you have defined the needs of your application integration environment and determined the capabilities that you require to meet those needs, you should investigate the technologies that you can deploy for application integration. This chapter examined how you can use various Microsoft technologies for application integration, and how these technologies map to the capabilities defined in the appendix.
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.