Guidelines for Application Integration
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.
Summary: This chapter introduces important application integration concepts and discusses the business and technical problems that an application integration solution can mitigate.
Welcome to Guidelines for Application Integration. Most organizations today use an increasing number of applications and services to solve specific business problems. In many cases, these applications and services exist on different platforms and were created at different times. The challenge that most organizations now face is to provide a method by which these applications can work together to address business goals that constantly evolve.
This guide examines in detail what application integration means and describes the capabilities needed to enable application integration. It discusses the major challenges involved and shows how you can adapt your application integration environment to meet those challenges. It also examines the Microsoft® software products and services you can use to help you design your application integration environment.
After reading this guide, you should be able to determine the requirements for application integration in your organization. The guide is not designed to show exactly which technologies to use in your integration solution. Instead, it focuses on the concepts that apply, regardless of the technologies you currently deploy in your environment.
This guide is written for technical decision makers and architects of organizations that are trying to determine whether they require application integration capabilities, that have already determined that they require application integration, or that want to improve their current application integration environment.
The guide does not provide a detailed analysis of how Microsoft products and services provide application integration; rather it discusses the issues at a more abstract and architectural level that applies to the products and services provided by any company. This guide does discuss, generally, how Microsoft can help you provide an application integration solution, but for more detail, you should examine the information available with each product.
Organizations of almost any scale may require application integration. Although application integration problems are usually more urgent in larger organizations, this guide makes no specific assumptions about the size of organization involved, or about the particular types of applications and services involved.
This guide does not require knowledge about any particular programming languages or products. To gain the most benefit from this guide, however, you should understand the issues and problems associated with designing and maintaining applications in an enterprise environment.
This guide is organized into chapters and an appendix as follows.
Chapter 1: Introduction
The remainder of this chapter introduces important application integration concepts. It discusses the business and technical problems that an application integration solution can mitigate, and examines some implementation choices for your application integration environment.
Chapter 2: Defining Your Application Integration Environment
To build an effective application integration environment, you need to understand the business problems you need to solve, and what you need from your application integration environment to solve them. This chapter outlines some of the important challenges and illustrates how your application integration environment can meet those challenges.
Chapter 3: Security and Operational Considerations
An effective application integration solution provides functionality that will rapidly become invaluable to your organization, so it is vital that you carefully design your application integration solution. This chapter discusses the security and operations considerations you need to examine to ensure that your application integration environment continues to operate in a reliable and secure fashion.
Chapter 4: Using Microsoft Technologies for Application Integration
Microsoft offers a number of products and services that can help you create your application integration environment. This chapter discusses the capabilities of these products and shows how you can use them in your application integration environment.
Appendix: Application Integration Capabilities
The appendix examines in more detail the application integration capabilities discussed throughout this guide. Use the appendix as reference to clarify how the guide defines these capabilities.
Your organization probably uses many applications and services that were built over many months or years, as new business needs were identified. As a result, these applications probably are of different ages, were written by different people using different languages and technologies, reside on different hardware platforms, use different operating systems, and provide very different functionality. In fact, many of your applications often have very little in common at all, resulting in isolated functionality and multiple instances of the same data. For your organization, these conditions can result in redundant activities, higher costs, and inefficient response to your customers.
If you have read this far, your organization has probably identified a business requirement for applications to work together. Just as employees have to work together to meet business goals, your applications need to do the same.
This guide defines application integration as follows:
Application integration is the secure and orchestrated sharing of processes and/or data between applications within the enterprise.
Note Although this definition restricts integration to sharing within the boundaries of an enterprise, the guidelines described in this document also help in providing integration between different enterprises.
Benefits of Application Integration
Effective application integration can provide your organization with the following important business benefits:
- Allowing applications to be introduced into the organization more efficiently and at a lower cost
- Allowing you to modify business processes as required by the organization
- Providing more delivery channels for your organization
- Allowing you to add automated steps into business processes that previously required manual intervention
Types of Application Integration
Application integration can be broadly categorized into three types:
- Manual application integration
- Semi-automated application integration
- Fully automated application integration
Most environments involve a combination of all three types. The following paragraphs describe each type in more detail.
Manual Application Integration
Manual application integration requires people (employees and customers) to act as the interfaces between applications and enable the integration between them. This form of application integration is very common. As an example, think of a customer service department that takes information from the public. People may enter the same information into multiple systems and read information from those systems to respond to customer requests. In other cases, a person may need to read customer information from one database, and then reenter it into another database used for another purpose.
This form of integration requires very little technology investment. It becomes more complex, however, when your organization becomes more complex, and can lead to inaccuracies in data. As the amount and complexity of your data increases, or as the number of applications increases, you will require more and more people to maintain such an environment. An environment that relies heavily on manual integration is generally very inefficient, and does not grow as easily as environments that use more automated techniques.
Semi-Automated Application Integration
Semi-automated application integration combines human steps with some automation. The person may be involved in an area where the corresponding automated solution is too difficult or expensive to implement, or where the business requires a person to make decisions. For example, your business may require a manager to approve all expense claims. In this case, all of the steps before and after managerial approval may be automated, but a person is required in the middle of the process. In other cases, human intervention may be needed to transform data that is required in another system.
Semi-automated application integration generally requires more technology investment, but once that investment is made, you can often reduce the number of people involved in integrating your applications. Reducing human involvement in this manner usually reduces costs and increases reliability.
Fully Automated Application Integration
Fully automated application integration removes people from the business process entirely, although they are required to maintain the solution. This type of integration consists of applications communicating through a series of interfaces and adapters. For example, two databases might share data, which is automatically transformed and committed to the second database from the first with no human interaction.
Although fully automated application integration removes the dependency on people, such systems can be more expensive to implement and may not be practical for many business problems. In many cases, you will still require people to make business decisions, and often it is more efficient to have a person control a technical process as well. For these reasons, you should decide where fully automated application integration is appropriate on a case-by-case basis.
Choosing the Right Types of Application Integration
Each type of application integration has its own set of costs and benefits, as outlined in Table 1.1.
Table 1.1: Costs and Benefits of Application Integration
|Application integration type||Costs||Benefits|
|Manual||Higher labor costs that scale badly. Subject to human error.||Little change required from existing low-technology environment.|
|Semi-automated||Higher technology costs to implement. Subject to design-time and run-time errors.||Lower labor costs. Scales better. Less subject to human error at run time. Faster processing.|
|Fully automated||Highest technology costs to implement. Subject to design errors.||Lowest labor costs. Not subject to human error at run time. Lose human decision making on business processes, but faster processing.|
Almost all environments involve a combination of all three types of application integration. Of course, in many cases people add to the effectiveness in your business processes. Situations where people often prove essential to effective business processes include:
- Business process exception handling. Many different events can lead to a business process exception. They can result from human error at some point in a process, faulty logic (in the applications or in application integration), concurrency issues, or errors in data received from business partners. Because it is not possible to estimate all the possible errors and their corresponding fixes, it is not possible to totally automate responses to business process exceptions. You need the ability to manually intervene in the business process in some cases and make the appropriate correction without being restricted by what the system is expecting.
- Manual override of business rules. Your applications will use business rules that are specific to the context of the enterprise at run time. These business rules facilitate automated execution of your business processes. However, you will need people to intervene in these business rules occasionally—for example, when shipping costs are waived for a large order from a regular customer. A fully automated solution with no manual override cannot allow for exceptions to be made based on human interaction with customers.
- Approvals at key points within a workflow. Workflows in your organization can often be automated as long as the values of selected parameters fall within predefined limits. Transactions with values exceeding these limits usually require human approval to make sure that they are reasonable. For example, even if the processing and approval of expense reports within a specified amount is automated, you should ensure that validation by appropriate management personnel occurs if the amount exceeds a predefined limit.
- System breakdown. A fully automated solution is completely dependent on the proper functioning of every component application that is part of the overall solution. Manual intervention enables you to temporarily replace a broken link in the chain of integrated applications with human transfer of information. For example, you may need to manually enter orders in applications that are ordinarily received by Electronic Data Interchange (EDI) if the EDI network you access is temporarily down due to a power outage.
If you closely examine processes that involve people, you can identify areas where semi-automated and fully automated business processes can be used. By doing so, you can reduce the number of employees involved in application integration and increase your potential for cost savings and scalability in your organization.
Making Application Integration Scalable
An important part of making application integration scalable is to increase the number of automated steps and reduce the number of human steps. This generally involves creating interfaces between applications along with predefined logic that replaces human involvement.
Increasing the level of automation generally increases the amount of information traveling between applications without increasing the number of employees required to support the environment. The scalability issues, however, do not stop at simple automation. You should also consider the number of applications themselves and how integration occurs between them. For automated application integration, you have two choices:
- Point-to-point model
- Integration hub model
The following paragraphs discuss each of these in turn.
The point-to-point model describes a decentralized structure in which each application communicates directly with the other applications. This type of integration is most appropriate for organizations that need to integrate a few applications with a small number of services.
Figure 1.1 shows the number of connections required for point-to-point environments with three and 10 applications, respectively, when you need to ensure that all applications can communicate with each other.
Figure 1.1. Point-to-point communication among applications
As the number of applications and services increases, the number of interfaces and connections that need to be maintained in a point-to-point environment rapidly increases.
The maximum number of connections required is defined as follows:
where n is the number of applications you need to integrate. The number of interfaces may be up to twice the number of connections (depending on whether they are one-way or two-way interfaces).
This means that for the three application example (Scenario A in Figure 1.1), up to three connections and up to six interfaces are needed, whereas for the 10 application example (Scenario B), up to 45 connections and up to 90 interfaces are required.
In a point-to-point integration environment, interfaces between the applications are usually written as business needs arise. The problem with this approach is the lack of consistency. The integration approach used generally depends on which integration approach the developer is most familiar with, what he or she knows about the individual applications, and so on. As more and more applications are developed or updated, additional interfaces must be created.
The same problem pertains when new business goals are defined that require applications to communicate with one another differently. Every change makes the environment more difficult to understand, until eventually the structure is so complex that it is almost impossible to manage effectively. Such a complex structure may even hinder the company's ability to strategically shift its business goals, due to the high IT costs surrounding the change. Defining application integration on the fly in this way can massively increase your medium-term and long-term costs for a short-term gain.
Integration Hub Model
The integration hub model provides a more centralized structure, in which an integration hub is placed between the applications, and each application communicates with the hub rather than communicating directly with other applications.
Each application needs only an interface and a connection to the integration hub. To simplify matters, the integration hub can rely heavily on existing standards, which means that either the interfaces already exist or the methodologies for writing them are well-defined.
The main advantage of an integration hub environment is scalability. Figure 1.2 shows the use of an integration hub, again in environments with three and 10 applications, respectively.
Figure 1.2. Applications connected through an integration hub
In Figure 1.2, a single connection is required to the integration hub, with interfaces potentially required at the application and the integration hub. However, many environments require only one interface (or none if the application already uses the standards supported by the hub). This configuration leads to a maximum of three connections and six interfaces for the three application example, or 10 connections with 20 interfaces for the 10 application example. In most cases, the number of interfaces will be significantly lower.
The integration hub model is significantly more scalable for integration environments with many applications. A typical large-scale organization has thousands of islands of information, involving thousands of applications. You simply cannot create individual interfaces for every point of interaction. Instead, the solution is to create an application integration environment that allows all of your applications to communicate in a logical, predefined way. This hub infrastructure enables you to modify or update elements much more easily, and to do so when the business requires, rather than when the preexisting technology dictates. It should also allow the organization to more easily change direction and to use the products and services it has to match evolving requirements.
Because interfaces are within the integration hub (and are usually standards-based), you do not need to rewrite them whenever new applications are introduced. However, the hub can be technically challenging to implement and may be too costly for more simple application integration environments. In addition, you may have to sacrifice some data complexity to ensure that each application conforms to the standards of the integration hub.
Choosing an Application Integration Model
As you examine the specifics of your environment, you will need to determine whether a point-to-point model, an integration hub model, or a combination of the two is most appropriate. Realistically, the decision you make will based on cost and value. In many cases, application integration involves a combination of both models, using point-to-point initially, and then moving to the use of one or more integration hubs as complexity increases and when there are clear business benefits to doing so.
Application integration isn't easy, hence the need for this guide. Typically, an environment that supports application integration meets at least the following requirements:
- Connectivity between different platforms
- Processing of complex business rules, including complex data transformation logic
- Support for business processes, from the very short to the very long, including processes that last weeks or months as data is passed and processed through different parts of the organization
- The ability to modify existing business processes or create new ones as business goals change
- The ability to adapt to changes in hardware, software, and business goals
To help meet these requirements, your application integration environment should:
- Expose a common interface through which applications can communicate, by using business semantics to request Web services.
- Allow service requests at the functional or data level for applications that do not support using business semantics.
- Use a common set of process and service rules to ensure consistency and reuse of integration services.
- Be capable of reusing the existing transport protocols that already exist in the enterprise.
- Insulate itself from existing technologies by using interfaces.
An application integration environment should not depend on the implementation of any particular technology. You don't know what applications and hardware your organization will be using in five years, but whatever they are, your organization will need to support them.
You can implement application integration in a wide variety of ways. Common choices include:
- Web services
- Extract, Transform, and Load (ETL)
- Communications message protocols
- Screen scraping
- Program calls
- Direct data access
- File transfer
- Human involvement
The following paragraphs discuss each of these choices in turn.
A Web service is an application component that:
- Exposes useful functionality to other Web services and applications through standard Web service protocols.
- Provides a way to describe its interfaces in detail, allowing developers to build client applications that talk to it. The description is provided in an XML document called a Web Services Description Language (WSDL) document.
- Describes its messages by using an XML schema.
Many vendors have endorsed and have started to provide Web service capabilities in their platforms. Using Web services for integration ensures that your integration is based on open standards that are language neutral and platform independent, and helps to ensure that the technology remains relevant over time.
Web services rely on defined specifications to allow systems from various vendors to interoperate. Web services specifications, commonly referred to as WS-* specs, are in various states of acceptance. The Web Services Interoperability Organization (http://www.ws-i.org) was formed to provide a framework and a set of tools to ensure that different vendors' implementations of a particular WS-* spec are compatible.
Note For more information about the current Web services standards being defined and the location of the specifications, see Web Services Specifications on MSDN® (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnglobspec/html/wsspecsover.asp).
Extract, Transform, and Load
Extract, Transform, and Load (ETL) is a data integration technology that allows you to consolidate critical information from multiple enterprise data sources. Data is extracted and transformed to fit the data model of the target data store and is then loaded into that data store. ETL technologies are typically used to load high volumes of information into data warehouses and data marts from multiple databases.
Communications Message Protocols
Communications message protocols, such as TCP/IP, HTTP, and FTP, define a standardized method of transport for messages exchanged between applications. They typically specify a standard set of headers as well as a way to package the content to be transported within the body of the message. Some protocols define the message headers for both the requests that the applications initiate and the corresponding responses. Your applications may use these protocols directly, or you may choose to build custom integration methods around the protocols.
Sometimes the business logic of an application resides in the presentation layer. In this case, you can use software packages that simulate the human user's interactions to interface with the application. Screen scraping is a noninvasive mode of integration and does not introduce any additional channels of interface with the application. You can implement this technology in two ways:
- Use terminal emulation software to intercept screen images after they have been formatted into the terminal data stream.
- Intercept the data before it is formatted for display purposes. This method generally provides better performance and requires less interaction than using terminal emulation software.
You can use program calls to access the internal resources of an application. Applications supporting program calls are typically multitiered and modular, permitting external programs to take the place of the internal user interface layer of the application. They have a predefined set of inputs (parameters) and outputs (responses) depending on their business context. If you modify these inputs and outputs, you may also have to make corresponding modifications within the applications that are already invoking them.
Direct Data Access
Direct data access involves making native database calls to application databases or writing data directly to the target application file systems. This approach requires knowledge of the data access language of the target database as well as knowledge of the data model used in the target application.
Direct updates to databases can make them vulnerable to data corruption and referential integrity violations. For this reason, many applications do not support direct data access. Instead, ordinarily you use their application programming interfaces (APIs) so that the calling programs can benefit from the business logic implemented within the applications.
File transfer involves integration between multiple platforms through transfers of files in batch mode. Typically, you move data between systems asynchronously using batch file transfers, and the flow of data is unidirectional. Changes in one system rarely affect other systems unless the structure, format, or semantics of the data being moved changes. File transfer is a cost-effective solution for transferring batch files between a wide variety of platforms. A number of features add to the reliability and efficiency of file transfer, such as scheduling, automatic restart, guaranteed delivery, encryption/decryption, and compression/decompression.
Almost every integration environment still uses people to some extent. People may be involved in one or more of the following activities:
- Presentation. Presentation is the extraction and subsequent presentation of the information to be transferred through various media to a person. Presentation is accomplished through fax and e-mail messages, printouts, or visual display of the information on an application screen.
- Validation. People are sometimes required to visually inspect data to ensure that basic business rules are being followed. The validation process can be augmented by providing the people with a predefined set of valid values (for example, reference codes).
- Transformation. People are sometimes required to reformat or make semantic changes to the data without making changes to the basic content (for example, changing date formats or regrouping information across address fields to fix data entry errors).
- Data entry. Data entry is the act of reentering the data into the target application. Basic cut-and-paste techniques can be employed to minimize the time spent and errors incurred in the data entry process.
Application integration poses its own special difficulties, some technical and some organizational.
Because application integration is a relatively new area of IT for most organizations, often no single product resolves all application integration issues. Instead, an application integration environment usually consists of a series of technologies, including prepackaged and custom software applications.
When you build an application integration environment, you are creating an environment that allows components to communicate with each other even though they were never designed to do so. In many cases, these components are not even aware of each other. They may include a variety of preexisting systems (some of which are essential to your environment), packaged applications with proprietary APIs, geographically disbursed databases, and a variety of hardware, operating systems, and protocols. Yet your environment needs to be noninvasive to existing applications. In these conditions, complex architectural issues such as the following can arise:
- Control and coupling
- Data exchange and data formats
- Distribution and concurrency
- Scalability, reliability, and availability
Some of the most significant technical challenges of designing an application integration environment involve identifying the technical needs for your solution and determining the combination of products and services that will provide for those needs. Because this segment of the market is still evolving rapidly, you should pay particular attention to the maturity of integration technologies, methodologies, and standards. You do not want to completely rebuild the application integration environment itself every few months. The aim, instead, is to have a relatively stable application integration environment, which is flexible enough to allow the addition of new applications or the alteration of relationships between applications.
Organizational issues in application integration arise mainly from the fact that your application integration environment will span multiple departments in your organization. Staff in different departments may choose to deploy applications that will then need to integrate with the rest of your environment. To design a truly effective application integration environment, you need to isolate application integration as a separate management function, and determine who is responsible for making it happen. In larger organizations, you may have a group of people specifically responsible for this function, including application integration architects, application architects, and application owners, whereas in other cases, application integration may be one of many management functions that you are responsible for. Either way, by defining the application integration management function in this way, you help to ensure that it is properly considered whenever changes are made to the IT infrastructure.
Establishing governance processes around application integration can be difficult, because the processes must encompass not only application integration, but also the end systems and business processes that are being integrated. In fact, during the deployment phase, you should be involved in defining the baseline infrastructure as well.
Organizational issues in application integration can be particularly tricky when you are working in a decentralized enterprise or in a business-to-business (B2B) scenario. However, these scenarios often have the greatest need for an effective application integration environment. Therefore, it is particularly important to resolve any organizational issues early on, so that you can focus on resolving the technical challenges of application integration.
An effective application integration environment should solve the problems of integrating applications, without requiring an enterprise to standardize on one hardware platform, one group of applications, or one operating system. Effective integration allows you to make a heterogeneous approach work even as you modify the elements within the mix. This chapter examined the problems that an effective application integration environment addresses, and discussed some of the implementation choices for application integration. The next chapter discusses what you need from your application integration environment to solve business problems.
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.