Guidelines for Application Integration

 

Appendix: Application Integration Capabilities

Microsoft Corporation

December 2003

Content

Business Processing Capabilities

Data Capabilities

Communications Capabilities

Security Capabilities

Operations Capabilities

This appendix provides a detailed description of the capabilities used to support an application integration environment. These application integration capabilities are divided into five categories:

• Business processing
• Data
• Communications
• Security
• Operations

Business Processing Capabilities

The capabilities defined in this section provide integration at the business processing level.

Orchestration

The Orchestration capability coordinates the execution of the different activities that make up a business process. Although a business process model is a static representation of the set of activities involved, the Orchestration capability brings the model to life and executes each step within the model. Activities undertaken by the people, systems, or organizations are invoked, coordinated, managed, and measured by the Orchestration capability, according to its process models.

The Orchestration capability triggers the processes corresponding to a business event, passes inputs and outputs across processes, receives communication of status information by different activities, and requests human intervention as necessary.

Event Processing

Generally, business processes begin either with the arrival of a message from the application integration environment communications capabilities, or they begin with the occurrence of a user action (for example, a user typing a customer's last name and then clicking the Search button in a Web interface). Both are considered events in the context of the application integration environment.

The Event Processing capability recognizes when business events have occurred, logs those events, and determines which business processing capability will receive the event. In most cases, this target is the Rules Processing capability.

The majority of events are expected occurrences, taking place as part of the normal business processing carried out by the environment. However, the Event Processing capability must also handle unexpected events, or exceptions. These exceptions are often the result of system-level events on the computers in the environment. For example, the system-level failure of a database that stores inventory may generate events at the business process level, because the business process of checking and updating inventory is no longer possible. In this case, the application could take tentative orders and warn the user that it is not possible to verify the availability of the product at that time. The same database failure might generate events at other levels as well. At the data level, the orders might be stored in a secondary location until the database could come back online, and at the operations level, actions would be taken to restore the database functionality.

Note   System-level events are not handled by the Event Processing capability; instead, they are handled by the Event Handling operational management capability.

State Management

In your application integration environment, you are likely to have a number of multistep processes, involving multiple applications that are interdependent. This situation can quickly become unmanageable, with a significant number of applications remaining active and simply waiting for responses from other applications that may not come for hours, days, or even months or years. These applications can consume significant processor resources. The State Management capability reduces the impact on your systems by maintaining the following types of state information across different components:

• Process state
• Message state
• Service state

Process State

Applications that use asynchronous messaging typically run until a message is sent asynchronously to a target application. At that point, the state of the multistep process is normally handed over to the State Management capability, and the application releases its resources and shuts down. When Event Processing recognizes the arrival of the response from the target application, it invokes the State Management capability, which in turn identifies the application that initiated the exchange and restarts that application so that it can receive the response to the original message.

To maintain the state of each of the pending applications, the State Management capability maintains a database with an entry for each pending process. For every pending process, the State Management capability has to store the data that must persist between process steps. Hence, State Management maintains a map of the various process steps with a pointer to the next step in the process (for example, the step that must be executed upon the arrival of the response to the last message sent).

Message State

Messages exchanged between applications are in various states of processing. Upon arrival, messages are in an active state waiting to be picked up and processed by the receiving application. After they are processed, they are in a journaled state. Messages can also enter the journaled state if they are in the active state for a duration longer than the predefined expiration period. This change of state usually occurs when none of the receiving applications picks up the active message. Journaled messages move on to an archived state when they are archived into a persistent store.

Service State

To run properly in its target environment, services require multiple pieces of information including user ID, time, date, contents, and any other information the service might need to access. A service that has all the required information is in a state where it is ready to run. Although most of the information is specified as input by the consumer, services may need to collect the remaining information by accessing other systems and data stores. While it is in the process of collecting this information, the service is in a state where it is preparing for execution. After execution, the service is in a dormant state waiting to be invoked by interested consumers.

Schedule

Not all pending processes simply wait for a response to a message. In some cases, a process is placed in the pending state until a specified time or date, or until a specified amount of time has elapsed. When this time has elapsed (or when the specified date or time is reached), the Schedule capability generates an event for the Event Processing capability just as any other event.

The Schedule capability also acts as a timekeeper, tracking processes where an external event should occur within a specified time frame or by a specified date or time. If this event has not occurred by the specified deadline, an alert or exception is generated and the Event Processing capability passes this alert to the process that was waiting fruitlessly for the expected event.

Business Transaction Management

Business applications are frequently required to coordinate multiple pieces of work as part of a single business transaction. As an example, think of when you order goods online. Your credit card must be verified, you must be charged, the goods must be selected, boxed, and shipped, and inventory must be managed. All of these steps are interrelated, and if one step fails then generally all of the corresponding steps must be cancelled. The Transaction Management capability coordinates these steps. In some cases, it is not possible to directly cancel out transactions that have already been issued. Instead, you may have to issue additional transactions to compensate for the original transaction. This technique is known as transaction compensation.

Rules Processing

Several types of rules may be applied in processing data—both to transform the data through the application of an algorithm (for example, a pricing model) and to route the data through application of a set of business rules (for example, orders for widgets are sent to the widget factory that is closest to the Ship To address on the order).

Some of the rules can be asserted at the time of development and require little access to outside systems or data. For example, all of the data needed to process the rule is contained in the message that is passed to the business capability. In other cases, the data needed to process a rule must be pulled from local and remote databases, and from local or remote applications.

The Rules Processing capability enables a developer or a business analyst to specify business rules without writing a lot of procedural code. The rules may be specified in a declarative programming language (for example, one that is not sensitive to arcane punctuation and does not rely on elaborate language constructs). Alternatively, the rules may be specified by answering questions in a graphical environment such as a programming wizard or a sequence of dialog boxes.

Your organization can define a wide range of rules within business processing capabilities, including:

• Data validation rules. For example, is this a valid customer number? Is the postal code valid for the city?
• Processing algorithms. For example, what is the total price of this order, considering the customer's contract discount and any additional discounts for the size of this order or the number of orders placed this month, and including the costs associated with the custom configuration?
• Processing rules. For example, was the customer's insurance policy in effect at the time of the accident? Or is the value of the order large enough to earn free shipping?
• Sequencing rules. For example, before the system can approve a second mortgage application, does it need to send a message to a scheduling application to have the house appraised?
• Exception handling rules. For example, if you don't have enough widgets in the warehouse, can you order stock from the supplier and have the the supplier ship the order directly to the customer?
• Non-delivery rules. For example, if the order isn't acknowledged by the primary supplier within four hours, do you resend the order, call the supplier's expediter, or send the order to a secondary supplier?
• Prioritization rules. For example, given the size of the requested insurance policy, do you escalate the request to a senior underwriter, or can a junior underwriter approve the policy?
• Data reconciliation rules. For example, since the last batch update of the customer database, the primary contact name for this account has been changed by both the customer service representative and the account executive—to different names. Which update takes precedence (or do you need to call the account executive for clarification)?

This list is not meant to be comprehensive, nor are the rules listed mutually exclusive. However, it is indicative of the range of rules processing requirements that must be supported in an integration solution.

Workflow

Often, business processing capabilities manage only the interaction between applications, because in many integration scenarios, user interaction is managed only indirectly. That is, the users interact with applications, and the applications in turn interact with the business processing capabilities.

For those processes that do interact directly with users, an additional business processing capability is needed: the Workflow capability. The Workflow capability uses the rules and state management engines to control the interaction between people and the process management environment.

For example, if an "application for fire insurance" event arrives at the process management environment, the Workflow capability may immediately forward the event to an underwriter, who decides whether the applicant should be insured and at what rate.

To ensure that the application is handled by an appropriate underwriter, a set of assignment and escalation rules may be defined and implemented with the help of the rules engine. Perhaps applications that pose no particular problems or risks can be assigned to the Junior Underwriters queue, whereas the riskiest applicants are assigned to the Senior Underwriters queue. In this example, the rest of the applications go in the Underwriters queue.

If the Junior Underwriters queue becomes too long, escalation rules can move up excess applications to the Underwriters queue. Similarly, if the Underwriters queue becomes too long, escalation rules can move up excess applications to the Senior Underwriters queue. However, rules do not permit the reverse to occur; for example, applications cannot be moved down from the Senior Underwriters queue to the Underwriters queue.

In addition to these escalation rules, the Workflow capability may implement prioritization rules. For example, if an application for fire insurance has been sitting in the Underwriters queue for two days, it can be escalated to the beginning of the Senior Underwriters queue so that it can be addressed before some of the new applications in the queue.

Contract Management

The Contract Management capability monitors and processes contractual obligations at run time. A business contract is an agreement between two or more parties expressing their mutual obligations and permissions for carrying out certain economic exchanges. These exchanges are related to the execution of actions that form multistep business processes.

In the context of business process management, a contract can be treated as a conversation between one party and other parties. The contract can be modeled as one or more workflows where the actions or tasks relevant to the contract (internal to organizations) are described as steps in the parties' business processes. The interactions between parties at various stages of the business process will often take the form of message exchanges.

In a service-oriented architecture, Web services can represent the parties involved in the contract. Specifications such as BPEL4WS can be used to address Web services orchestration and coordination issues. WS-Policy is the emerging standard that is used to implement contract agreements.

Data Capabilities

The capabilities defined in this section provide integration at the data level.

Schema Definition

Schemas describe the structure and format of data. Each schema contains property information pertaining to the records and fields within the structure of data. This information is vital to ensuring that the data actually makes sense. Fortunately, in many cases the schema definition already exists and you only need to maintain the definition, rather than redefine the structure of the data.

Schema definitions are presented in a number of different forms in different applications. These forms include well-formed XML, document type definitions (DTDs), Electronic Data Interchange (EDI), and structured document formats. Many predefined schemas in each of these formats may be appropriate for your environment, depending on the applications that you use. For example, hundreds of schemas are associated with the dozens of EDI documents that are defined within each of the multiple versions of the two EDI document standards (X12 and EDIFACT).

You may find it useful to preload schemas from standards organizations, if the schemas are likely to be used in your environment. Sources of such schemas include:

• RosettaNet
• Open Applications Group (OAG)

Other frequently used schema definitions are available from the e-marketplace, including:

• Gas Industry Standards Board (GISB)—used by the energy industry
• Transora—used in e-commerce
• WorldWide Retail Exchange—used in business-to-business scenarios
• Covisint—used in the automotive industry
• Exostar—used in the aerospace and defense industries
• ChemConnect—used in the chemicals, feed stocks, and plastics industries
• FreeMarkets—used in supply management
• E2open—used in process management

In current applications, one of the most commonly used schema definition languages is XML. However, because it is fairly likely that your environment will include applications that do not use XML, you generally should support multiple schema definition languages. Doing so is important in any case to ensure that you support future technologies.

It is also possible to automatically generate XML schemas from data structures such as relational databases, fixed and variable format files, application programming interface specifications, COBOL copybooks, and so on. As well as being much easier, automatically generated schemas are less prone to errors.

Often templates are used for defining common data or documents formats, such as purchase orders, invoices, and shipping notices. The following is an example of a purchase order schema definition:

 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://tempuri.org/po.xsd" 
xmlns="http://tempuri.org/po.xsd" elementFormDefault="qualified">
 <xs:annotation>
 <xs:documentation xml:lang="en">
  Purchase order schema for Example.com.
  Copyright 2000 Example.com. All rights reserved.
 </xs:documentation>
 </xs:annotation>

 <xs:element name="purchaseOrder" type="PurchaseOrderType"/>

 <xs:element name="comment" type="xs:string"/>

 <xs:complexType name="PurchaseOrderType">
 <xs:sequence>
  <xs:element name="shipTo" type="USAddress"/>
  <xs:element name="billTo" type="USAddress"/>
  <xs:element ref="comment" minOccurs="0"/>
  <xs:element name="items" type="Items"/>
 </xs:sequence>
 <xs:attribute name="orderDate" type="xs:date"/>
 </xs:complexType>

 <xs:complexType name="USAddress">
 <xs:sequence>
  <xs:element name="name"  type="xs:string"/>
  <xs:element name="street" type="xs:string"/>
  <xs:element name="city"  type="xs:string"/>
  <xs:element name="state" type="xs:string"/>
  <xs:element name="zip"  type="xs:decimal"/>
 </xs:sequence>
 <xs:attribute name="country" type="xs:NMTOKEN"
   fixed="US"/>
 </xs:complexType>

 <xs:complexType name="Items">
 <xs:sequence>
  <xs:element name="item" minOccurs="0" maxOccurs="unbounded">
  <xs:complexType>
   <xs:sequence>
   <xs:element name="productName" type="xs:string"/>
   <xs:element name="quantity">
    <xs:simpleType>
    <xs:restriction base="xs:positiveInteger">
     <xs:maxExclusive value="100"/>
    </xs:restriction>
    </xs:simpleType>
   </xs:element>
   <xs:element name="USPrice"  type="xs:decimal"/>
   <xs:element ref="comment"  minOccurs="0"/>
   <xs:element name="shipDate" type="xs:date" minOccurs="0"/>
   </xs:sequence>
   <xs:attribute name="partNum" type="SKU" use="required"/>
  </xs:complexType>
  </xs:element>
 </xs:sequence>
 </xs:complexType>

 <!-- Stock Keeping Unit, a code for identifying products -->
 <xs:simpleType name="SKU">
 <xs:restriction base="xs:string">
  <xs:pattern value="\d{3}-[A-Z]{2}"/>
 </xs:restriction>
 </xs:simpleType>

</xs:schema>
  

Schema Recognition

The Schema Recognition capability checks that the schema can be read and understood, and that it is well-formed. The well-formed XML standard dictates that an XML document have a single root and that elements must nest completely or not at all.

Because it is likely that multiple formats will be used for different schemas, it is important that your schema recognition capability be able to accept multiple schema formats. After a schema is recognized and registered, subsequent data can be received and checked for compliance to the corresponding schema.

The Schema Recognition capability also performs schema validation. Schema validation is the verification that the schema definition conforms to a predefined document structure (for example, a DTD).

Mapping

Data is often rendered in different formats in source and target applications. The relationship between two applications and how the data must be transformed from the source to the target is handled by the Mapping capability.

As you define your mapping capability, you should consider both the development (or design-time) component, and the run-time component. For development, the mapping capability should enable developers to specify the mapping logic in a declarative, nonprocedural way—for example, using the mouse to delineate the mapping between a field in the source data structure and the position of that data in the target data structure. At run time, the mapping capability should be able to access reference data (for table lookups) and perform other data enrichment functions, such as sorting and filtering multiple input records before creating an output data set.

Two forms of mapping can be performed:

• Field mapping
• Semantic mapping

The following paragraphs discuss each mapping type in turn.

Field Mapping

Field mapping is a data translation process in which the records and fields in the source specification are related to their corresponding occurrences in the destination specification. When data is moved between different applications, several levels of transformation may be required.

In some cases, each field that is moved from the source to the target representation has to be converted. Maybe the source application allows mixed case, but the target application accepts only uppercase. Or maybe the source application allows additional characters (such as dashes or spaces) for formatting purposes.

Often, the data can be converted by applying a straightforward conversion algorithm (for example, temperatures can be converted from Celsius to Fahrenheit through a simple calculation). In other cases, though, the transformation may require a table lookup and replacement (for example, converting the gross weight of a product to net weight may require a lookup based on the product code).

Semantic Mapping

As data is moved from one application to another, the data structure itself may need to be modified. This modification is known as semantic mapping.

As an example of semantic mapping, consider the different ways of representing addresses. Table A.1 shows two separate applications storing the same address information, but in a different structure.

Table A.1: Different Ways of Storing Address Information

Moving data between these two representations requires shuffling the data elements around a bit, but care must be taken so that none of the semantic values of the data (in other words the meaning of the data) is lost as a result of this reorganization of the data elements.

Data Validation

Data requirements for an enterprise change over time. You may have preexisting systems in your environment that produce data that is not of the quality that the business now requires. Or a change in business context may mean that data that was once correct is now incorrect. The Data Validation capability can verify that data meets the criteria you set and is therefore useful to transfer between applications. The complexity of the validation depends on the enterprise business rules and the capabilities of the application integration tool itself. However, Table A.2 gives some examples of what to expect.

Table A.2: Example Data Validation Rules

Data Transformation

The Data Transformation capability is the capability that actually renders the content of input data elements to the corresponding elements of the output data as specified in the map. Data Transformation works in conjunction with the Mapping capability to ensure that not only is data sent to the correct location, but that it is in the right form when it arrives. The Data Transformation capability performs a number of important tasks, including:

• Data aggregation/disaggregation
• Data enrichment
• Data summarization
• Data filtering

The following paragraphs discuss each of these tasks in turn.

Data Aggregation/Disaggregation

Some data must be merged together before it can be sent. Data may need to be combined from multiple applications, or from a single application over time. The Data Transformation capability uses the Mapping capability to identify the data to be merged and composes new data out of elements of the input data. The map tells the Data Transformation capability where the data elements come from.

Data aggregation may be required to perform both transactional (one record-at-a-time) transformation and batch, set-oriented transformation. For nearly real-time or real-time transactional transformation, the Data Transformation capability may wait for the arrival of messages from several applications and then combine that data to create a single output message. Or, the Data Transformation capability may receive a message and then more proactively request the additional data elements from other applications.

Data aggregation is often constrained by the Rules Processing capability, which monitors the composition process and examines the integrity of the composed message.

Data disaggregation is the opposite of data aggregation. Specific data may need to be broken up into several pieces of output data. Semantic mapping specifies the output data, and data disaggregation decomposes the input data into the output data.

Data Enrichment

When input data is being formatted into output data, the input data may not hold all of the information that is required for output data. The Data Transformation capability enables you to specify where to acquire the information to enrich the output data. In many cases, this information is acquired from a data store. The Data Transformation capability may use the Data Access capability to look up this information source.

Data Filtering

The Data Transformation capability provides a mechanism with which users can filter out information from specific data. There may well be an occasion when it is either difficult to filter the information at the source, or when the filter should only be applied to certain targets. Applying the filter to certain targets is very important when you want to filter out sensitive information that a business user does not want to send to certain partners. Data filtering is also used to disseminate predefined cross-sections of data from source applications to target applications across multiple channels. Using data filtering can prevent you from having to change the source application, which may save significant effort and maintenance costs.

Data Access

As data is being passed from application to application, often different collections of the data are required by each application. One important consideration is how you take the data that already exists and optimize the process of accessing that data. There are three choices to consider for data access:

• Dynamic data access
• Staged data access
• File/database access

The following paragraphs discuss each of these data access choices in turn.

Dynamic Data Access

Integration solutions are often best served by dynamically creating business objects, where a business object contains all of the required data to be accessed by a business process. The Data Access capability creates these business objects when source data is published or when a request arrives from a composed or straight-through processing (STP) application.

Staged Data Access

In some occasions, creating data for access dynamically isn't the best approach. For example, when the data collection process has an unacceptable impact on the response times of the applications that are being accessed (specifically, the response times for the regular users of the source applications), the integration solution has to be adjusted so that the impact on these users is less troublesome.

Similarly, response times for the STP or composed application may be unacceptable. For example, suppose that you are building a composed application and that collecting the data required to meet users' needs will involve sequential access to ten applications, each with a response time of two seconds. The two-second response time for each individual application is quite acceptable to most users. However, the cumulative response time of 20 seconds is not acceptable.

These performance issues can be addressed by placing the data into a separate database, called a staged data set. A platform for staging data (using some intermediate data format) is useful for those integration scenarios where the data from the source applications can't be accessed directly and immediately from the source applications.

Putting data into a staged data set is a straightforward process when the data is stable and when the access from the integration programs is read-only. However, if the source applications update the data on a regular basis, those updates have to be propagated from the source applications to the staged data set within a short time frame. Otherwise, the data will become stale and less useful to the integration programs. The reverse problem is also a possibility. If the integration programs are going to change the data in the staged data set, then those changes have to be propagated to the source applications in a short time frame.

File/Database Access

In addition to accessing data from applications, data stored in flat or structured files as well as databases need to be retrieved for processing. Most file and database access mechanisms are programmatic.

Communications Capabilities

The capabilities defined in this section provide integration at the communications level.

Request/Reply

The Request/Reply capability facilitates synchronous communication between applications. In request/reply interactions, the source application sends a request to the target and then performs no other processing until the reply is received.

Request/Reply does not normally use queuing because the requestor is waiting for a reply. Instead, a timeout parameter is set. If the reply is not received in a certain amount of time, the requester resends the request.

Request/Reply also uses other capabilities in the environment to provide its functionality. For example, the Directory capability can resolve addresses for the Request/Reply capability.

To enable request/reply interaction between two applications, a common data format is required as well as common data and process semantics.

Connection Management

When a request/reply interaction takes place, the Connection Management capability establishes a logical connection between the requesting and the replying applications. Connection Management assigns a connection ID to both applications and then returns the reply to the application that sent the request.

Message Routing

The Message Routing capability controls the routing of messages from source to target applications. Message routing is a quite simple problem when it entails moving the data from Point A to Point B. In other conditions, however, the routing process can be quite elaborate—for example, when moving the data from Point A to a hub, where the message may be forwarded to Points B, C, or D, based on a variable such as time of day, server load, or message content.

The message itself is a formal construct containing the data. You can think of the message format as an envelope, with the contents of the envelope being routed based on the addressing information that is listed on the outside of that envelope. In practice, the addressing information is often placed in a message header and the content of the message then follows the header in the message's data structure.

The routing functionality is built into the message hub, which inspects each message in sequence, determines where to send the message, and then sends it to the target system.

Addressing

Your application integration environment cannot route messages correctly unless it can determine where to send the messages. The Addressing capability maintains the information about where to route the message. The environment can determine the location of the target system(s) through either direct or indirect addressing.

Direct Addressing

Direct addressing is used when the source application specifies the target application for communication. The source application may either specify a network address or a name, that the Addressing capability in turn resolves to a network address, through a call to the Directory capability.

Indirect Addressing

Indirect addressing occurs when the source application does not specify a target application. Instead, the Addressing capability determines the correct target application after the message has left the source application.

There are two forms of indirect addressing: publish/subscribe addressing and content-based routing.

Publish/Subscribe Addressing

In publish/subscribe addressing, the source application sends out the message without indicating the target application; the identity of the target application may not even be known to the source application or its developer. Instead of a destination address, the source application identifies each message by providing a topic name.

Independent of the activities of the source application, potential target applications register with the Addressing capability when they are initialized. During this registration, each target application indicates that it is interested in receiving all messages on a specified topic.

Publish/subscribe addressing provides a useful anonymity to sources and targets, meaning that applications can be added or removed at any time. At run time, the messages are published by the source application(s) with topic names, and the Addressing capability builds a routing table automatically to route each message to the target application(s) that had previously subscribed to messages about the topic. Published messages are often kept in a data store for access by subscribers.

Content-Based Routing

Content-based routing is closely related to publish/subscribe addressing. In content-based routing, the source application again does not specify the target application's identity. However, content-based routing differs from publish/subscribe addressing in that a developer builds a routing table that specifies rules to determine the correct match between a particular message and a target application. For example, the table may ensure that all orders for a certain product go to the factory that makes that product.

Message Forwarding

Sometimes a message must be passed through several applications before a response to the original request can be completed. The Message Forwarding capability handles such serial messages.

The forwarding capability can also serve another purpose. In some integration implementations, several intermediaries are needed to provide message management and monitoring capabilities. In such cases, Message Forwarding moves the data between each of the intermediaries, while keeping track of the required sequence.

Message Delivery

The Message Delivery capability determines how messages pass from source application to target application. Message Delivery occurs through one of two mechanisms: multicasting or single casting.

Multicasting

Message delivery through multicasting involves delivering the same message to all (or nearly all) of the potential target applications and letting them decide whether to keep the message or not.

The multicasting approach may seem rather wasteful of bandwidth. In reality, however, most local networks are Ethernet-based, which works according to a similar principle, so the multicasting approach does not lead to excess traffic on these networks. In fact for transmission over local area networks (LANs), multicasting is often the most efficient form of message delivery.

Of course, the vast majority of networks consist of multiple LANs. In situations where the message is intended for addresses on other LANs, gateways are used to forward the multicast message to the LANs where the remote addresses reside. Most integration scenarios involve the bridging of several LANs with message delivery gateways.

Single Casting

Message delivery through single casting involves delivering each message to specific targets as identified by the Addressing capability. Single casting is not always the most efficient way of sending messages, because addresses have to be resolved before they can be sent to each recipient, which can slow performance. However, single casting is very useful in some circumstances; for example, when a number of intermediaries perform routing, filtering, and forwarding functions on the message.

Message Queuing

In application integration, it is often important to provide guaranteed delivery of each message to ensure that the message reaches its recipient. It is also important that each message is delivered once and only once to its destination. Otherwise, it would be easily possible, for example, for a customer to receive multiple shipments of goods when he or she placed only one order. Finally, it is important that the messages arrive at their destination in the order in which they were sent. This ensures that, for example, an order correction does not arrive before the order that it is correcting.

The implementation of reliable messaging depends on the Message Queuing capability. This capability stores each message in a queue until it can be successfully delivered.

This queue can be implemented in memory or on disk. Implementing this queue in memory can maintain high performance, but at the risk of losing messages if the server fails. Disk implementations trade off higher reliability for slower performance because disk I/O is required every time the message metadata has to be updated to reflect completion of another step in the guaranteed delivery process.

When a message is not delivered successfully, it remains in a queue, and the Message Delivery capability attempts to redeliver the message. After attempting delivery a set number of times at a set interval (as defined by an administrator), the Message Delivery capability places any messages that cannot be delivered in a dead letter queue. Administrators can inspect the dead letter queue, and correct a message addressing header or message content to ensure successful retransmission to the intended destination.

At this point, a message may also have to be sent to the source application to indicate this failure. Or, a message may be sent to an administrator who can help determine the source of the addressing error.

Message Instrumentation

Although messages are sent in a specific order, they can be received in any order. The Message Instrumentation capability helps ensure that the messages are processed in the order that was originally intended.

The correct order can be ensured by using sequence numbers on the messages themselves. In this case, Message Instrumentation adds the sequence numbers and accepts messages only if the previous message has been correctly received. It may also add the name of each intermediate hub to every message along with a time stamp as it reaches each hub. This additional information enables you to create historical reports on messages and may assist in resolving any performance issues.

Message Correlation

In an asynchronous interaction, there is no logical connection between the sending and receiving applications. When the receiving application replies to the initial transmission, the receiving application has to specify the proper application name or network address along with the reply message.

Because an application may send several messages asynchronously before receiving a reply for any of them, the application that sends asynchronous messages uses the Message Correlation capability to match up or correlate each reply with the corresponding transmission. Each time a message is sent or received, the Message Correlation capability is called so that it can append a tracking number to the message header. In terms of functionality, this mechanism is very similar to the Connection Management capability used in synchronous messaging.

Serialization/Deserialization

Data often exists in a hierarchical format in applications. However, as it travels between applications, data must be converted into a flat structure so that it can be sent over the network. The Serialization capability does this conversion. Of course, after the data arrives at its destination, it must also be converted from a flat structure back into its original hierarchical structure. The Deserialization capability does this conversion.

Transactional Delivery

Often, you must guarantee the delivery of messages. The Transactional Delivery capability groups together messages, sending and receiving them within a transaction. This means that such messages either are sent together in order (a committed transaction), or are not sent at all (an aborted transaction). Likewise, transactional messages are only read (removed) from a queue if and when the transaction is committed. Otherwise, they are returned from a queue and can be subsequently read during another transaction.

The Transactional Delivery capability ensures that transactions meet the following requirements, known collectively as ACID (Atomicity, Consistency, Isolation, and Durability), which are defined as follows:

• Atomicity refers to the need to complete all the parts of the transaction or none of them. For example, when a pick list is generated to take a book from the shelf, the inventory quantity for that book should be decreased.
• Consistency refers to the need to maintain internal consistency among related pieces of data. For example, in a funds transfer application in a bank, a $100 withdrawal from a customer's savings account should be completed successfully only if a corresponding $100 deposit is made in that customer's checking account.
• Isolation refers to the need to ensure that the activities within one transaction are not mixed with the actions of parallel transactions. For example, if five simultaneous requests for a particular book arrive and the warehouse has only four copies of the book in stock, the five transactions must each run in isolation so that the orders are processed properly. Otherwise, it might be possible for each of the five business transactions to see an inventory level of five books and then try to reduce the number to four.
• Durability refers to the need to ensure that the results of the transaction are changed only by another valid transaction. The results of the transaction should not be compromised by a power outage or system failure.

To assist in providing durability, your transactional delivery mechanism could employ a disk-based delivery method, where every message is written to permanent storage as it moves through the system. However, to provide durable transactions, the message must be committed to a persistent storage the moment the transaction is completed.

Note   To overcome system unavailability, you can use persistent messaging as an alternative to Transactional Delivery. Persistent messaging makes the message itself more resilient.

Transactional Delivery has a performance cost, because of the overhead involved in ensuring that delivered messages can be rolled back, or in resending messages after a delivery failure. However, the performance tradeoff is often necessary to provide additional guarantees during message delivery.

Batching/Unbatching

You may not always have continuous, fast, and reliable network connections throughout your enterprise. It can therefore be very useful to collect messages and then send them when the link is available. The Batching/Unbatching capability provides this functionality.

One of the disadvantages of a batching and unbatching approach to communications is that it can lead to unacceptable inconsistencies in data. For example, the same object can be deleted from a database in two separate locations, but there may be no realization that this has occurred until the batch processes take place later.

Encode/Decode

Different operating systems use different character encoding methods for representing text characters. For example, IBM mainframe applications use Extended Binary-Coded Decimal Interchange Code (EBCDIC) as a character encoding method, while other systems either use ASCII or Unicode. The Encode/Decode capability is responsible for ensuring that applications can still communicate, despite using different encoding methods.

Even when two systems use the same encoding method, there may be differences in the actual encoding due to the use of different code pages. These different code pages are necessary because many languages use different alphabets. Some vary in minor ways (for example, Spanish uses a character constructed by placing a tilde above the character "n"). Others are radically different (for example, the Romance and Germanic languages use the Latin or Roman alphabet, whereas languages such as Arabic, Hebrew, Japanese, and Korean use entirely different alphabets).

When communication capabilities receive the data, the Decode capability is invoked as necessary to convert the data to the encoding method and code page used by the components of the integration solution. This decoding ensures that integration functions such as transformation and routing can be properly performed. After the integration middleware is ready to transmit a message to the target system(s), the message may have to be recoded by using the encoding system and code page of the target system(s).

File Transfer

In some cases, rather than using request/reply or message-based communication, applications must communicate by exchanging files over the network. The File Transfer capability handles this process of moving or transmitting files between applications.

File Transfer can be useful in an environment when applications are not designed to communicate with other applications directly, but are designed to read, manipulate, and create files. You may also use the File Transfer capability in conjunction with the Batching/Unbatching capability to collect messages into a file, and then transfer that file at a particular time or date, or when a reliable communications link to the target system is available.

Although file transfer lacks some of the sophistication of other communication capabilities, it is still a widely used communication mechanism, due to the number of applications that read and write to files as their only way of communicating. One advantage of file transfer over messaging-based communications is that this type of communications usually is possible within a simpler infrastructure than messaging.

Security Capabilities

The capabilities defined in this section help provide security to your application integration environment.

Authentication

As a user or system attempts to connect to another system, the user or system generally presents a set of credentials. The process of verifying those credentials is known as authentication.

Authentication is one of the most fundamental aspects of IT security, because it verifies the identity used to connect, and the security context of the connection. There are many possible methods of authentication, but the most commonly used today is a name and password combination. This method is popular because it is quite quick to implement; however, because this type of authentication is generally custom implemented, the level of security and interoperability really depends on the design and implementation.

To increase the security of authentication itself, it is necessary to encrypt the credentials through a network authentication protocol. The Kerberos authentication protocol is commonly used because it is supported by both UNIX and the Microsoft® Windows® operating system (Windows 2000 and later). Kerberos is a network authentication protocol, defined by the Internet Engineering Task Force (IETF), that relies on public key cryptography. However, the use of Kerberos for authentication with third parties requires trust relationships to be established and access to the Kerberos key distribution center. The Kerberos key distribution center is where the private keys of principals (users or systems) are kept for encrypting information.

For more information about the Kerberos protocol, see RFC1510 (http://www.ietf.org/rfc/rfc1510.txt).

Authorization

Authorization is the process of determining whether a particular connection attempt should be allowed. It occurs after authentication, and uses the identity confirmed in authentication to determine if a connection attempt should be permitted.

Authorization can be used to restrict the systems that can request integration capabilities. Such restriction is useful if your organization requires strict control on which systems can use its capabilities. A number of existing authentication protocols, including Kerberos, also provide authorization capabilities. Kerberos relies on a ticketing mechanism; after authentication, the principal is provided with a unique ticket. This ticket can have an access control list (ACL) attached to it. The ACL is used to determine the resources that the principal can access.

Identity Management

It is quite common for a single user or system to have multiple sets of credentials that are used to access different systems. The Identity Management capability allows you to manage these credentials and map them to the correct user or system.

Identity Management is particularly important for effective application integration, because it allows single sign-on (SSO). This means that if a user needs to be authenticated, the sign-on should be required only once per session, with the appropriate credentials being used to authenticate the user in other environments.

Single sign-on for user authentication is an important convenience for users and can also save time in application integration scenarios by reducing the amount of human interaction. Identity Management is equally important when applications are making the requests, because it ensures that the correct credentials are passed to the capability. Without an identity management system, this programming logic would need to be built into each application.

The two common identity storage implementations for a single system rely either on a Directory capability application or on custom application logic, which in turn relies on a relational database system.

Note: Identity management functionality can be provided out of the box with an identity management product. Custom implementation of this technique requires proper security threat analysis to prevent security holes through which unauthorized users or applications can access a capability by assuming the identity of an authorized user or application.

Security Context Management

Authentication and authorization allow a user or application to communicate with an integration capability. However, when that capability accesses the underlying applications, it must in turn provide credentials to those applications. This management of credentials is achieved by the Security Context Management capability.

Security Context Management provides credentials to applications through two methods:

• Impersonation. The integration capability impersonates, or assumes the identity of, the original requester when accessing the underlying applications. The actual credential used for authentication with the back-end applications can be the original credentials provided, or the Security Context Management capability can obtain the appropriate credentials from the Identity Management capability.
• Consolidation. The integration capability uses a single identity to access the underlying application. Consolidation grants the same permissions to all requesters that can enter the capability. Consolidation is commonly used for composed applications because it provides simple user management to the existing systems and a better opportunity to pool connections. Connection pooling can improve system performance and is commonly implemented for database access. The benefit of connection pooling is minimal when the request is quite small, compared to the overhead of setting up a connection to the existing application to initiate the communication. However, when you need to transmit a significant amount of data across a network, pooling connections can improve performance dramatically.

Profile Management

Profile Management manages principal profiles. Generally, applications use principal profiles to provide customization that is specific to the principal accessing the system. The profile management system can store and manage information such as user roles. Given that the Identity Management capability keeps track of the principal identity, the system commonly stores additional information, such as profiles for the principal. This allows the profile information to be extracted at the time of authentication and authorization, which in turn allows efficient use of resources.

Information Protection

There are essentially three ways that systems can protect information:

• Encryption
• Hashing
• Obfuscation

The following paragraphs discuss each of these information protection methods in turn.

Encryption

Data is encrypted to prevent unauthorized users from easily viewing or tampering with it. A common encryption technique relies on a secret key system, in which the sender uses a secret key to encrypt the data, and the recipient must provide a corresponding key to decrypt the data. In the context of application integration, secret key encryption is probably the most commonly used form of encryption.

One of the biggest issues with secret key encryption is the requirement for the sender and recipient of the data to know the same secret key. The more parties involved in the communication chain, the more places to which the secret key must be distributed and the greater the risk that the key may become tainted. The encrypted data is usually tied to the exact key used to encrypt the information. If the key is changed, existing encrypted data must be decrypted and reencypted with the new key.

Another method of encrypting data is known as the public key system. A public key system uses two types of keys: private and public. The private key is kept by the owner of the key, whereas the public key can be widely distributed. Data is encrypted using the public key, and the same information can only be decrypted using the private key. Parties wanting to use public key encryption must have the recipient's public key to be able to encrypt the data. Another important feature of this system is the ability to sign data. Signed data can be viewable during transmission, so it is not encrypted. The idea behind signing is to ensure that the actual data has not been altered. Signing essentially uses the hashing technique on the data.

Public key encryption relies on digital certificates, which are files containing a unique (cryptographically generated) sequence of letters and numbers. Public key encryption uses a public certificate and a private certificate representing the public and private keys, respectively. The use of and access to the private keys usually requires a password, so in effect there are two layers of protection. The downside of public key encryption is that it requires additional infrastructure support. This infrastructure, commonly referred as public key infrastructure (PKI), consists of the following protection mechanisms:

• Hashing. Hashing, also known as one-way hash, generates a hash value for data. This hash value provides a unique identification for the information, similar to fingerprints. Hashed information cannot be retrieved from the hash value, because the hash value does not carry the original information. Identity management systems commonly store the hash values of passwords so that the actual user password is never stored and so that no one other than the user knows the real password value. The system simply hashes the password given during authentication; if the hashed value matches the hash value stored, then the user must have provided the same password value because hash algorithms provide statistically unique hash values.
• Obfuscation. Obfuscation describes a protection mechanism using a simple algorithm that does not rely on user-defined keys. In effect, the information is encrypted, however, if the algorithm is known, any user can decrypt the information. Obfuscation is normally used to protect program codes. Higher-level languages such as C++, the Microsoft Visual Basic® development system, Java, and C# ultimately compile to machine code. Java and Microsoft .NET compile to a more user-friendly code that makes reverse engineering slightly simpler. Obfuscation attempts to make it difficult for reverse-engineered programs to be easily understood.

Note   For key based encryptions and hashing, the algorithm used can be well known. However, the data is still protected because it relies on unique keys to protect the data. Obfuscation, on the other hand, does not rely on keys; it relies on the obfuscator algorithm to secure the data, which means that when the algorithm is known, the information can usually be easily decoded.

Nonrepudiation

Nonrepudiation describes the ability for a party to be identified without ambiguity through digital signatures. The idea behind nonrepudiation is similar to the use of signatures on credit cards. Credit card signatures are meant to allow the authorized user to verify his or her identity and, more importantly, to prove that the user authorized the charge.

Credit card fraud is common because thieves can forge the cardholder's signature. In the digital world, nonrepudiation currently relies on public key encryption, although it has been suggested that a combination of public key encryption and biometrics would enhance security. It is important to stress that for public key encryption the level of security increases with the size of the key or the number of certificates. Currently, the most common key size is 128 bits. It is estimated that to decode 128-bit encrypted information using the most powerful computer today would take 1 trillion years. Obviously as computers become more powerful, the key size will need to be increased to provide additional protection.

Technologies such as Digital Rights Management essentially use the public key encryption mechanism of data protection to provide nonrepudiation. Digital Rights Management controls the access and distribution of digital information. However, no consistent and universal system currently exists that allows particular information to be restricted to certain users for certain periods of time. For example, when an e-mail message is sent, the information it contains is outside the sender's control. The recipients can copy or forward the message to anyone. Digital Rights Management can help by allowing the sender to restrict copying or forwarding to other people who have not been given explicit rights to view the message.

Operations Capabilities

The capabilities defined in this section provide operations functionality to your application integration environment.

System Monitoring

System Monitoring includes the monitoring of hardware, the operating system, and software applications to ensure that these various system components are functioning correctly and within the desired operating parameters or agreed-upon capability levels.

One of the most fundamental and often ignored aspects of monitoring is the concept of baselining. Every system behaves uniquely in terms of how quickly it responds to certain requests and the amount of resources it consumes to satisfy requests. It is very important to obtain baseline values for each of the metrics being monitored. Baselining establishes a standard against which future comparisons can be made to detect anomalies.

There are several ways of viewing what a baseline really means, depending on the maturity of the system being monitored. For a newly commissioned system or a system receiving a major upgrade, it is useful to be able to obtain a baseline value based on a single transaction for each of the different transaction types. As the system matures, you may find it helpful to establish a different set of baselines based on other metrics that make sense for your application. For example, you may create a new baseline, measured only on working days to reflect the fact that systems exhibit certain load characteristics based on the working days of the week. Baselining can also be extended to the provisioning of capabilities and the associated time and cost involved. Baselines can help you to estimate the cost and time involved in extending the existing capability (through scaling up or scaling out) to handle larger loads.

Event Handling

Event handling can be classified into two categories: passive and active.

Passive event handling is commonly found in applications. It basically receives the event and logs or display the information. The system does not attempt to understand the exception type and resolve it.

Active event handling requires the capabilities of passive exception handling, but provides the additional capability to process and attempt to resolve the issue that caused the exception.

A resilient integration application provides some capabilities of active event handling. This allows minor exceptions to be automatically handled and allows the requests to go through as normal. A very simple example is a system that tries to access another back-end system through an unreliable network connection and as a result may lose requests. The integration application can build a timeout mechanism such that it can automatically retry if the first request fails.

Business Activity Management

Business Activity Management is an important aspect of any integration system. However, the term Business Activity Management is broad. In the context of this discussion, you should consider Business Activity Management as the monitoring, managing, and analysis of business transactions processed by the integration system.

Activity monitoring enables real-time tracking of business transactions. Alerting of business exceptions is an important feature of monitoring because it enables business analysts to react to any problems with business transactions as they occur. Fixing problems before users notice them, or being able to at least notify users in advance of potential delays, is a key part of proactive monitoring. Examples of problems can range from systems being unavailable to particular steps of the process exceeding the maximum allowed processing time. The latter is important for businesses that must adhere to business capability-level agreements with their customers or partners.

Activity management enables a business analyst to react to a business exception by modifying the transaction process path in real time, before the system rejects the transaction. Actual modification of predefined business processes is considered part of the development and application release process, due to the more permanent nature of the changes.

Activity analysis is about capturing all of the business transaction information that has passed through the system. All data that is relevant to the business transaction—such as process step response times, business data values, and data sources—is captured and is placed in a data warehouse. This allows the business analyst to perform historical analysis of the business process steps from numerous aspects. This ability is extremely important for process-oriented businesses that have capability-level agreements with customers or partners. Having this information allows the business to quickly verify whether it has breached any capability-level agreements. For example, imagine that a parts supplier has agreed with various business partners to process any orders and provide an invoice along with delivery tracking within three days from the time the order is received. If the integration system handles the whole process, it will be able to capture the duration of any process steps as well as the end-to-end processing time. Each step may involve other systems, or it may involve humans who approve or perform further processing. If the partners complain that the business constantly misses its processing-time commitments, the business can then perform an analysis and determine the reason for the delays. Similarly, having the right monitoring in place allows warnings to be generated if a particular step has exceeded its maximum processing time.

Configuration Management

For the purposes of this guide, configuration management refers to the tracking of hardware and software configurations. Although most configuration management tasks focus on process and documentation, a number of technology capabilities can assist in this area.

Note   This guide does not cover asset management.

Configuration management is important for a number of reasons. One of the most important is that it helps you keep track of how each system is configured. This information is very advantageous to have for disaster recovery scenarios. Another benefit of configuration management is that it helps you to compare two systems and to verify them before and after you make modifications, such as application upgrades. It is quite common for systems to exhibit stability issues for no apparent reason, but then on investigation to reveal that some of the files have been replaced or modified to different versions.

Being able to track dependencies is another important part of configuration management. Properly maintained dependency information can provide a real benefit in the area of change management, especially when determining possible impacts of a particular change and what regression testing might need to be performed.

Versioning

Versioning forms an important part of configuration management. Versions allow you to quickly verify whether the application or operating system files are correct. They can also help you to determine whether your applications have backward or forward compatibility.

Note: Effective versioning requires technology and policy support. It is important to ensure that version guidelines are defined and followed.

It is possible for applications to detect version information and enforce policies that determine which particular version of libraries can be used with the current version of the application. This mechanism ensures that either the application runs as expected or that it fails right away, if the required version of the libraries is not available.

Signing the actual binary images in conjunction with versioning provides an additional layer of protection and verification, which can be important when ensuring if certain files have been modified due to virus infection or malicious acts.

Some of the most commonly changed information is configuration data, which administrators often use a text editor to change. Signing the configuration file allows changes to the configuration information to be detected.

Integration applications may also keep a recent history of the configuration file, or upon loading a new configuration file may generate a log highlighting the changes between the newly loaded version and the version currently used. This mechanism allows specific attributes to be tracked and easily identified, simplifying troubleshooting problems related to errors in configuration.

Automated Provisioning

One of the most common issues with manual configuration is human error. It is the main reason that large organizations create images or automated system provisioning systems to reduce the amount of human involvement. Systems are usually composed of a variety of applications, software patches, and hardware. Even when the same applications are installed on two systems, if they were installed in a different sequence, some aspects of the configuration may be different. Such situations can produce problems that are difficult to discover. Automated provisioning of the base system and applications helps to ensure consistency between systems and platforms.

System Configuration Snapshot

A system configuration snapshot provides the ability to capture a snapshot of the system when changes were made. Such a snapshot essentially provides a mini backup capability, which allows the system to be rolled back when the changes produce undesired results. Without a system configuration snapshot, the usual course of action is to revert to a backup of the system.

Using the snapshot information, you can also compare the system to provide a before-and-after view and to track and control exact changes.

Configuration Verification

It is very important for you to be able to identify and verify the configuration of your systems .If you cannot, configuration of any system is at the mercy of the quality of provisioning and change management. When something goes wrong, it is difficult to check whether the configuration has been modified in any way.

Change Management

For the purpose of this guide, change management is limited to the considerations of change management, and does not specify the process itself. If you are interested in a detailed process-oriented view of infrastructure management, you should refer to IT Infrastructure Library (ITIL) and to the Microsoft Operations Framework (MOF).

Change management is particularly important for effective application integration. Any integration application relies on a number of other applications, each of which is susceptible to changes from its respective owner. Changes to these applications are likely to have a significant impact on the integration application itself. Because the sheer volume of changes required can be large, having effective change management processes and tools helps to reduce the chance of errors when applying changes. Change management can also help to reduce the amount of time required for changes to be applied.

An important goal of having a change management system and process is to allow controlled changes to occur in the shortest period of time. Controlled change is an important criterion in maintaining a stable system. Without controlled changes, a system modification can produce undesirable results.

There are a number of items to consider when you establish your change management procedures:

• Release management. Release management involves planning how the change will be released as well as determining rollback or roll forward plans. Rollback entails reversing the changes and ensuring that the system is restored to the exact same configuration it was in prior to the release. Roll forward is necessary when the changes are known in advance to be irreversible. In such cases, it is vital to have plans in place to mitigate the risk of the system not functioning correctly. Ultimately, due to the possibility of frequent changes that can occur in an integration application, providing as much automation as possible will help speed the implementation of changes and reduce the risk of human error.
• Regression testing. Regression-testing the new application ensures that it does not compromise existing functionality. When the regression testing methodology and steps are well-defined, automating this process can help during release management and can reduce human-related errors.
• Backward compatibility. Backward compatibility is especially relevant in the integration area. Backward compatibility allows the functionality of the integration application to be enhanced or modified without affecting existing systems that relied on the application.

Directory

The Directory capability is essential in the context of integration, because it stores much of the information required by application integration.

Generally available directory systems are highly scalable and often reside across multiple physical servers. As a result, normally a delay occurs before changes to data are consistently presented across all physical servers. Hence, if the data stored in the directory changes often—a few times a minute, for example—the directory may never present a consistent view of that data across all physical servers. For this reason, directories normally provide fairly static information that changes a maximum of a few times a day. Application interaction with the directory is largely restricted to read-only interactions.

The directories that are used in an application integration environment include:

• The identity directory, which is primarily used for security as the repository for identities and related information.
• Subscription directory, which is primarily used in a publish/subscribe scenario.
• Application configuration, which is primarily used by the applications themselves and stores configuration information.
• Capabilities directory, which provides a list of capabilities offered.

The identity directory is the repository of all user and system identities. It may also contain other related information such as the user roles and profile information. Many organizations aim to have a unified identity directory, but struggle to do so, due to the number of preexisting applications and the custom developed security mechanisms they use. The problem is exacerbated due to the number of different environments the applications are using.

The subscription directory, as the name implies, keeps track of subscribers. This directory is typically used in publish/subscribe communication. A separate directory ensures that the subscriber base can be managed outside the application logic, which allows for either self-subscription or total management of subscribers, or for a combination of both.

The application configuration directory is most useful in providing a consistent application configuration across a number of systems. For example, if an integration system currently requires 10 different servers and relies on a local configuration file, the operator must ensure that all 10 are consistent. Providing an application configuration directory allows these settings to be provided in a single location. Although there are tools that can assist to maintain the files, using them introduces another level of complexity, and there may be limitations in how the tools can replicate the configuration files to the servers.

The capabilities directory provides a common place where capabilities can be listed and used. In the case of Web capabilities, Universal Discovery Description and Integration (UDDI) servers provide this functionality. Regardless of whether the capabilities directory is a UDDI server or a custom server, it is important to ensure that the information reflects the type of capabilities provided in a commonly understood format and protocol. Commonly understood, in this case, can mean organization-wide or even spanning across to partner organizations.

Start | Previous