Leveraging the Value of Your Host Systems with Real World SOA

Authors: Chris Kabat & Bill Topel, MPS Partners

Published: April 2007

To download a printer-friendly version of this document, click here.

When implementing a service-oriented architecture (SOA), it makes sense to take a gradual and pragmatic approach. When IBM host systems are involved within the architecture, it makes even more sense to minimize risk through an approach that realizes results quickly. This document walks the reader through an example of implementing a real-world SOA by using Microsoft® BizTalk® Server 2006 and the BizTalk Adapters for Host Systems.

Starting with a concrete business problem, this article attempts to walk the reader through many considerations and different approaches available when integrating with IBM host systems. Using the example of electronic gift cards for a retailer, the document continues to show how SOA can be achieved quickly by using Microsoft technologies. The document dives deep into different mechanisms for accessing an IBM host system and provides some working examples using BizTalk Server and the BizTalk Adapters for Host Systems.

This document is targeted at a wide audience consisting mainly of architects and developers. However, business decision makers would also gain some insight by understanding the process and considerations of moving to service-oriented architecture.

As enterprises adopt service-oriented architecture, it makes sense to choose a gradual and pragmatic approach. This approach is especially effective when attempting to leverage existing assets on a host system such as an AS/400 or mainframe computer. Years of business rules, processes, and critical data live wrapped in resources that can be supported only by fewer and fewer technologists, and usually at a higher cost. Trying to unwind this logic and potentially replace these systems with more modern technologies that allow for interaction with information workers can be quite difficult and may take years.

By taking a service-oriented approach, with middleware products such as BizTalk Server 2006 and the BizTalk Adapters for Host Systems (Host Integration Server 2006), some of the business rules and business processes can methodically be migrated to a new platform without affecting existing processes. Opportunistically, processes that are most critical to the business can be migrated first. Overall, this can help organizations achieve higher operational efficiencies, increased trading partner communication, and a higher level of overall customer satisfaction.

An organization can lower their risk when leveraging host system assets by doing so in a gradual manner using a number of different approaches as defined below:

  • Opportunistically expose important business logic with a small amount of host development, if necessary, to make programs "callable." Then use the appropriate tools to expose these processes as services. In essence, extend service orientation to the host environment.
  • Provide enterprise integration via data access, queues, and direct host file access.
  • Take a very non-invasive approach to expose business logic via screen scraping technologies. This can be accomplished through utilities provided by a number of Microsoft partners. (See http://www.microsoft.com/windowsserver/midrange/partners.mspx for more information about partners related to IBM AS/400 migration.)

By using service-oriented techniques, consumers of the legacy assets will no longer know about or depend on the implementation of those assets. As the enterprise evolves, those services may continue to access the host system, or they may just implement the interfaces of a newly purchased line-of-business system. Either way, the enterprise has made the decisions that will allow them to remain agile and change with their customers' needs in the long term. The decision to begin this approach cannot be taken lightly, and should be driven by the needs of the business.

To demonstrate the benefits of an opportunistic approach to service-oriented architecture, it makes sense to frame the discussion in the context of a real-life example. This example is based on a common business scenario faced by a large apparel retailer. This retailer had a large number of brick and mortar stores and outlets, as well as a very successful catalog sales channel. Over the past ten years, they quickly grew a strong and dynamic e-commerce presence as well. The company operated their business on a number of different systems, developed over a period of more than twenty years. A large number of those systems utilized a host environment. This IBM AS/400 environment was primarily accessed via terminal emulation and printed green bar reports.

As the business expanded their interfaces with new point-of-sale equipment, third-party co-packers and shippers, and online Web channels, a number of new requirements and challenges faced their expanding business. Their e-commerce channel continued to become a much bigger factor than their catalog channel. The constant change and competition in the e-commerce world introduced a need for agility that they were not prepared for. To accommodate these changes, new custom Web applications were continually architected and implemented. These applications required more modern payment methods and practices such as prepaid gift cards and other safe-payment mechanisms. To make things even more complicated, these new requirements typically needed to interact with the existing business logic and data that was located on the legacy host systems. New interfaces and new technologies were required to accomplish the new requirements. Over time, the number of point-to-point interfaces continued to grow. It became more and more difficult to manage. This can be seen in the diagram below:

Figure 1 The Business Problem
Bb507734.5a376c67-8e42-4e33-a004-be87a889dcb4(en-US,BTS.10).gif

Faced with all of these challenges, the retailer had some complex decisions to make. They could write individual interfaces for each new requirement and continue to support them into the future, or they could embrace a broader standard architecture based on modern principles such as service-oriented architecture. During the conversations that ensued, some of the following observations were made:

  • New government regulations, such as Sarbanes-Oxley, required tighter control around security, data access and user control, and transaction validation.
  • Current IT best practices and technologies allowed information to be accessed in real time without complete replication of data and security structures.
  • Web services, based on standard interfaces, can be securely delivered over the public Internet. This type of technology was being demanded by more and more trading partners.
  • Newer technologies allowed host transactions, business logic, and data to be more readily accessed via Web services.
  • A new breed of information technology employees had joined the organization over time, and there seemed to be a cultural divide beginning to form between the more seasoned host system programmers and the new developers educated in object- and service-oriented techniques.

In the end, the retailer decided they needed to take their architecture to the next level to best service their customers. A decision was made to explore how the design and architecture would look using modern design and development techniques. Although it was still a very important part of their organization, the retailer decided to decrease new development on their AS/400 host system. They needed an architectural approach that would allow them to accomplish this goal over time at a minimal risk.

As the retailer looked for the most efficient solutions to their recent challenges, members of the team continually were referred to concepts of designing and implementing a service-oriented architecture (SOA). Service-oriented techniques allow organizations to create loosely coupled systems that provide for a large amount of both reuse and agility. One of the keys to successful SOA is breaking apart complex problems into smaller domain-specific problems that can be more easily designed and managed. Instead of creating monolithic applications, SOA defines smaller services that can be brought together through composition and orchestration to solve those complex business problems. Through contract-first techniques, discussions of platform and protocols are usually left as an implementation detail. Contract first means that a discrete contract, usually defined as an XML schema, is created to define interfaces before any implementation is executed. Both the service and the consumer are able to develop against that contract independently by using this approach. Intersystem dependencies, which can usually be the biggest risk, were defined up front and early in the design process.

The key principles of SOA and contract-first development allow organizations to take an opportunistic approach to adoption of the technology. Existing systems can be wrapped with services that implement a well-defined contract. These services can now be consumed by some of the more modern tools that help define and manage complex business processes and business rules. As the organization matures, the services themselves can be replaced with more modern development techniques and technologies, can be moved to a different platform, or can be implemented with a commercial off-the-shelf system (COTS). Because the contracts of the services have not been changed, the consumers can stay the same. This gives an organization the opportunity to create a bigger-picture SOA landscape without affecting existing processes. This modular and more gradual approach is much lower risk and can provide the ability to show the overall value of SOA in a much shorter period of time.

The retailer and their architecture team decided that SOA and contract-first development techniques were the best way to add agility and efficiency to their organization. However, there was a large concern that these modern techniques would not fit within their current environment. They wondered if it was really possible to leverage decades of host development by using these techniques. They were unsure that the tools available would allow them to access the data and business processes that they needed to make this initiative successful. It was time to take a deeper look at their existing host systems and verify that they could start this migration in months instead of years.

As the retailer looked at their options with service-oriented architecture, it made sense to look at their biggest unknown. They were apprehensive that their host systems, and the developers that support them, could be brought into this new architecture. As with other companies that create plans to move toward a service-oriented architecture, they asked themselves what role the IBM host would play in their environment. As they built their service-oriented architecture road map, they realized that many companies continue to keep their IBM host as a major component of their architecture. Most companies running a host computing environment rely heavily on the platform for transaction processing, key trading partner information exchange, and third-party application integration. To that end, unless the plan is to migrate away from the host system near term, putting the system into the architecture makes sense.

When initially reviewing an SOA, some of the decisions that need to be addressed are how to access application business logic, how to authenticate internal and external users, how to manage application access and control, and how to implement a secure, scalable, and cost-effective solution. Sometimes the high cost of transactional processing on a host system environment is justified by the high risk of moving away from the architecture due to mission-critical systems.

After the retailer made the decision that the IBM host system would play a role in their SOA, the most likely scenarios were examined. In a service-oriented environment, the four most likely integration points are:

  • Access to host-based business logic.
  • Access to host-based databases.
  • Access to host-based security hierarchies.
  • Access to host-based transactions.

Most service-oriented architectures will need to provide all of the system functionality outlined above. The real question, however, is what functionality will reside on the host.

Another key question is how, and at what pace, a company will migrate toward SOA. It is clear that many firms are opportunistically integrating only the required applications to the new computing model. When moving to SOA, the big bang theory is a poor approach because it takes too much time to show success and business value. The opportunistic approach allows companies to choose only the relevant applications and at the relevant times. Most companies have realized that not all applications need to migrate to a service-oriented environment. In fact, after reviewing applications and user profile groups, only a portion of a company’s total applications need to integrate to the new environment.

To that end, the first decision is what applications will be integrated with the platform and what applications will continue to run on the platform natively. The next decision that needs to be made is, for the applications that need to integrate with the host, what and how the interoperability solution will be built. The final and most complex question is, now that a future state architecture exists, what is the best way to adopt this new strategy while managing the risk.

While the above decisions are being considered, several areas need to be evaluated. In most host environments, traditional emulation, host printing, security, data access, and communications methods will need to be reviewed when defining the future state. Additionally, the best approach for separating the business and data access in legacy COBOL or RPG programs will need to be thought through.

Although host languages such as COBOL and RPG are procedural in nature, they can be implemented in a way that lends them very well to service orientation. Through callable areas such as PLISTs in RPG and the communications area in COBOL, programs on host systems can be passed parameters, executed, and return parameters. Unfortunately, these parameters (or structures) are not typically passed back and forth in a standard format. Microsoft BizTalk Server 2006 and the BizTalk Adapter for Host Applications help to solve this problem. This suite of tools has a feature called Transaction Integrator that allows you to execute RPG and COBOL programs directly. These tools also convert data from XML and .NET data types to host data types.

Accessing programs directly is just one way to access an IBM host system. Other mechanisms that can be used to access host data and business rules are:

  • Reading and writing to DB2 repositories via the BizTalk Adapter for DB2.
  • Receiving/transmitting data to and from the host system by using MQSeries and data queues.
  • Directly reading and writing to host files by using the BizTalk Adapter for Host Files.
  • Screen scraping techniques using third-party software.

The retailer began to realize that with all these different options at their disposal, the ideal of SOA in the short term quickly began to become a reality. They decided the best way to begin was to take the first step and create their first service.

The retailer described earlier had a specific problem that needed immediate attention. The retailer had an electronic gift card management system on a host system. This system was far from open and could only be accessed by other host programs. The retailer realized that to stay competitive with other chain retailers, they needed to provide these gift cards through other channels such as local grocery stores. A third-party electronic gift card clearinghouse came to them to offer their services in this market, and it seemed like it would be a great relationship. Because this retailer had other channels that used gift cards, they wanted to be able to accept any gift card purchased from any channel and then use it via any channel (for example, buy a card at a brick and mortar store but use it online or through their customer service for phone orders). This meant that they needed to maintain the status of gift cards in a centralized point. They had a working gift card system that worked across their Web and customer service channels. They also wanted to provide these gift cards in time for their holiday season and the quickest way to do that was to reuse their existing infrastructure.

The retailer knew that the clearinghouse was working with some existing enterprise application integration techniques with their other customers, but opted to take a different approach. They agreed on a contract for electronic gift card management with the clearinghouse in the form of XML schemas in and out. They also agreed on a number of operations they would support for the clearinghouse. The retailer had defined their first service to implement.

Before focusing on the current problem at hand, the retailer chose to take a step back and define a high-level roadmap for their SOA initiatives. Although they knew that this future state might be years away, they felt it made sense to look three to five years in the future and see where they wanted their architecture to be. To accomplish this, they took an inventory of all of their existing systems throughout the enterprise and interviewed analysts and users of each of those systems. This included order management systems, e-commerce systems, warehouse management systems, coupon and gift card systems, point-of-sale systems, marketing systems, sales data warehouse systems, and many smaller ancillary systems.

Once the interviews were complete, the retailer categorized each system into logical business domains (such as order management, payment, and inventory). The interdependencies and redundancies in each of the source systems were analyzed. Key modules that were used across the enterprise were separated into a corporate shared domain. A domain was also created for utility services. Utility services were more technical services that were shared across the domain, such as e-mail capabilities, logging, and exception handling. The retailer diagrammed these domains and created a future state diagram that they could use as a reference during their new initiatives. They also decided that they would designate “Source of Truth” systems for each of their types of data. For example, the AS/400 was the source of truth system for gift card information and tax tables. Referring to Figure 2 below, the items highlighted in yellow are the pieces of the architecture that were addressed as part of this discussion.

Figure 2 Future State SOA Architecture
Bb507734.ff32f1db-a9e2-4dc7-a13c-61d0858a279a(en-US,BTS.10).gif

The next step was to define a set of guiding principles that they could use to steer their SOA initiatives. These principles were simple rules to help keep them on track with their long-term goals. For example, choosing to buy over build when cost effective was a guiding principle. Another principle was to define a service as a corporate shared service whenever it was used by more than one system. They chose to put a higher emphasis on reuse on these corporate shared services and set a higher bar when it came to change management and governance. These corporate shared services were also required to be hosted using Microsoft BizTalk Server 2006 to ensure loose coupling and provide a platform that helped with security, operational monitoring, and scalability.

The concept of the service life cycle was also addressed as part of the architecture phase. The retailer defined manual governance processes to track services as they were created, needed to be changed, or needed to be decommissioned. These processes and procedures were embedded in their current software development life cycle (SDLC) documentation. They knew that this was a short-term solution and that as the number of services grew within their organization, they would need an enterprise tool to help manage these processes.

Once the future state enterprise architecture was defined, the retailer took an opportunistic approach by choosing to begin their SOA initiative with a single corporate shared service. They began with the gift card service in the payment business domain. As described previously, there were a number of challenges with how they currently handled gift cards using their host system.

Figure 3 Future State Gift Card Service
Bb507734.5f91ca38-d018-48de-84cf-ea0e9dee1a49(en-US,BTS.10).gif

They decided that this new gift card service, used by multiple channels within the enterprise and with multiple external trading partners, would provide the following operations:

  • Activate: Given a predefined gift card number (or set of numbers), this operation marked the gift card as ready for use. They reduced a point of fraud by requiring an activate step as part of the gift card purchasing process.
  • Deactivate: Given an activated gift card number, this operation marked the card as unavailable for use. This was usually done when there was a problem with the gift card or when the purchase was stopped post activation but prior to the end of the customer’s total transaction.
  • Validate: Given a gift card number, this operation returned the state of the gift card and the amount of available credit.
  • Pend: Given a gift card number and amount, this operation marked that amount used.
  • Commit: Given a gift card number and amount, this operation deducted the amount from the gift card. This was usually called at the end of the order process.

Knowing that multiple systems across the enterprise needed to do some development to be able to call the new gift card service, the retailer went with a contract-first approach. This approach gave them the ability to start developing the changes to the consuming applications before the service was complete. For each operation, a request and response schema was created. The request and response schemas for the activate operation are portrayed below:

Figure 4 Activate Request/Response Schemas
Bb507734.0e6ae291-eed9-4afa-8619-9b0e2ec59b27(en-US,BTS.10).gif

It is worth noting that some information about the requesting system was included in the request schema. The design team decided this would be convenient information even though they knew they would have other mechanisms, such as digital signatures, to handle party resolution. The response schema had a standard error block that allowed consumers to handle the exceptions from each corporate shared service in a similar way. By providing these “standard” request and response blocks, they felt they were giving themselves some common ways to track each of their corporate services. They knew they could intercept this information and provide service-level data by using a tool such as BizTalk Activity Monitoring (BAM). BAM allows data from within messages (or data about messages) traveling through BizTalk to be intercepted and tracked. Based on user configuration, this data is stored in a repository and potentially in OLAP cubes for fast and simple data analysis. This data can also be fed from other applications by using an API provided by BAM. BAM also provides an out-of-the-box portal to allow business users to view this data. In short, BAM provides an end-to-end solution for real-time analysis of complex business processes hosted in both BizTalk and non-BizTalk environments.

One of the retailer’s guiding principles was to implement all corporate shared services within Microsoft BizTalk Server 2006 by using a standards-based approach. The best approach for the desired architecture at the time was Web Services Enhancements for Microsoft .NET 2.0 (WSE). (In the future, BizTalk Server 2006 R2 and the Windows® Communication Foundation (WCF) adapter will be a much better choice.) Given the request and response schemas, a service proxy was created by using the Microsoft WSE Publishing Wizard. Each operation was defined and the request and response schemas were specified as shown below:

Figure 5 WSE Web Services Publishing Wizard
Bb507734.e9ad087a-9d08-45c7-b748-afbf9a1d7c3b(en-US,BTS.10).gif

As an output of this wizard, a .NET Web service proxy was created. A BizTalk Server 2006 receive port and location combination were also created. These artifacts worked together so that when a consumer called the service, the request message was routed to the BizTalk Server message box and a corresponding response was routed back. The loose coupling provided by this approach is important to note. The request is published to the message box and the response is returned based on a different subscription. Those subscriptions are created via a process called binding in BizTalk Server. The actions of the service are a detail of the service implementation. The service could be processed via an orchestration that composes other services. The service could be a binding to another service using content-based routing techniques. The contract is in place and that is what is important. Now service implementation and consumer implementation can happen somewhat in parallel.

Now that the contract was defined, the focus was on the implementation of the service. Using the opportunistic approach, different approaches were used to implement different operations of the service as described below.

The retailer decided the best way to mitigate the risk of taking on the new SOA techniques was to take the existing gift card system as is. This system was hosted on an AS/400 and written in RPG. They did not want to modify any host code if possible. This was mainly done to avoid affecting any of the existing channels using the gift card modules on the host system while the gift card service was being developed. To allow this to occur, they decided to implement their services through screen scraping techniques.

Screen scraping techniques emulate the actions of a user at a terminal emulation session to execute applications on a host system. These techniques can be very fragile in the long run because they depend upon fixed screen layouts and flow that can change over time. Screen scraping also can cause scalability issues since it is dependent upon emulation sessions and is almost always single threaded. However, screen scraping can allow the solution to be created without any host system development at all. Basically, the service will act like it is a customer service representative and hand-enter the gift card information into the gift card system and look at the appropriate response on the screen. These techniques may not be considered the best way to accomplish the task at hand, but may be necessary if host system development is not a possibility. This usually occurs when the required functions exist in programs that implement both the user interface logic and the business logic.

In this example, customer service was able to access a gift card and activate it through a single screen via a 5250 session. A C# object was created using third-party screen scraping libraries to log on and activate the gift card. A response on the screen was read to determine the success of the activation. To remain loosely coupled, this code was implemented as an operation of a Web service. The retailer’s guiding principles stated that corporate services were to be implemented by using a middleware platform, so the retailer decided to implement them via a BizTalk orchestration. In fact, the retailer knew they would want to add some additional capabilities in the future and wanted a solution that would allow them to compose some additional services. Within the BizTalk orchestration (shown in Figure 6), a request message was built for the SSActivate operation by looping through the consumer’s original request message. The responses from the SSActivate operation were used to build the response to the end consumer.

Figure 6 Activate Operation
Bb507734.9d2f2497-6d1f-4ed5-ae9f-8237f02d7af4(en-US,BTS.10).gif

The deactivate and validate operations were implemented with a similar pattern using screen scraping techniques. Since the pend and commit operations were done in batch programs during order processing and did not have screens available, they needed to be implemented using a different approach.

The retailer now had a working gift card service with activate, deactivate, and validate operations, but they wanted to implement the pend and commit operations. After further analysis into the order-processing programs, it was noticed that the order-processing programs called a single DB2 stored procedure called CHGCRDST to change the status of the gift card and handle the appropriate auditing logic. The decision was made to use the DB2 adapter to call this stored procedure from BizTalk orchestrations.

To be able to access the DB2 stored procedure from the pend and commit orchestrations, the first step was to generate the appropriate schemas for use by the DB2 adapter. This was done by using the Add Adapter Metadata Wizard in Microsoft Visual Studio®. This wizard can be accessed by right-clicking the BizTalk project and selecting Add >> Add Generated Items. The DB2 adapter is included as part of the BizTalk Adapters for Host Systems. The wizard prompts for information including the DB2 connection string, and whether a stored procedure, updategram, or select statement should be used. The connection string is used to query the DB2 database for objects that can be selected. The wizard also requests namespace and request and response root element names for the schema used to communicate with the DB2 adapter. After running the wizard and giving it appropriate information about the CHGCRDST stored procedure, the following schema was generated:

Figure 7 CHGCRDST Request
Bb507734.c8c2f2e9-bc6e-4e7e-88cc-d5697ad28a93(en-US,BTS.10).gif

An orchestration was created to execute the CHGCRDST stored procedure. The orchestration iterated through each pend request and built a DB2 request message using the previously defined schema. This request message was sent to a logical port named DB2Port, which in turn responded with an appropriate result that could be interrogated to verify that the action succeeded. The orchestration created is pictured below:

Figure 8 ChangeStatus Operation
Bb507734.c1b8fcda-b716-4f6a-b820-e90671494934(en-US,BTS.10).gif

It also is useful to look at the configuration of the physical port that the DB2 logical port shown above was bound to. The physical port uses the DB2 adapter, and the advanced configuration is as shown:

Figure 9 DB2 Adapter Configuration
Bb507734.4727b637-536c-40e2-9405-7ef4cca69543(en-US,BTS.10).gif

When configuring the adapter, a connection is defined to the DB2 database. As can be seen in Figure 9, there are a few properties that must be configured for this adapter. Clicking the button for the connection string property launches a wizard allowing the Microsoft .NET Data Provider for DB2 connection string to be created. One option within this wizard is to pass a username and password. It is generally best practice to configure the connection string to use Enterprise Single Sign-On (SSO) as opposed to hard coding a username and password. The target namespace and root element of the document being returned should also be configured and the URI will be automatically populated based on the connection string.

This same process was repeated for the commit operation. All of the operations of the gift card service had now been implemented.

There was another item that came up as part of the design. As can be seen from the orchestrations pictured, if an error occurred a SOAP fault was generated to the consumer. On the host system, these types of errors were tracked in a central location. The retailer was satisfied with this error-handling mechanism, for now, and wanted to make sure that the gift card service could alert this process if any errors occurred. The decision was made to not have two exception-handling processes or error repositories across the enterprise. This error-handling mechanism accepted an error message in a fixed-width text format on a data queue, asynchronously.

To accomplish this error-handling capability, the same fault message that was passed to the consumer was to be passed to the data queue via an MQSeries server. In the send port that sent the message to the MQSeries queue, the following steps were taken:

  • The fault message was mapped to a flat file schema using a map on the send port.
  • The flat file schema-based message was converted to a fixed-width text representation of the SOAP fault using a flat file assembling pipeline component embedded in a custom send pipeline.

The creation of the flat file schema and map is beyond the scope of this discussion. The send port to the MQSeries was configured as shown in Figure 10:

Figure 10 MQSeries Adapter Configuration
Bb507734.61482bb8-b8f2-48c9-b312-a5624a6d23fa(en-US,BTS.10).gif

This example sent the message to a queue on a Windows-based MQSeries server. This queue was configured to transmit the messages to the AS/400.

The changes to the orchestrations were fairly negligible. The exception handlers were modified to pass the fault message to a logical port called MQErrorPort, which was in turn bound to the physical send port described previously.

The retailer now had the gift card service in production available for use by both their internal systems and external trading partners (such as the gift card clearinghouse). However, they still faced some challenges:

  • The activate/deactivate/validate operations used screen scraping and could be considered fragile if any screen changes were made. Performance and scalability were also concerns.
  • The pend/commit operations shared a DB2 stored procedure with the order-processing batch programs. There was a concern that additional business rules might be executed in the batch programs, and in the future this would most surely occur.

The decision was made to create two reusable programs on the host system by reorganizing some existing parts of the order-processing system and gift card programs. These programs were:

  • GFTCRDI: This program, given a gift card number, would return some key information about the gift card. This could be used to validate the gift card.
  • GFTCRDCS: This program, given a gift card number and optionally an amount, would change the status of the gift card. The program would return success or failure. This could be used to activate, deactivate, pend, and commit the gift card.

The programs were created and the existing gift card programs and order-processing batch programs were modified to use these new programs. The next step was to re-factor the gift card service to use these new methods instead of screen scraping. This implementation was accomplished by using Transaction Integrator and the BizTalk Adapter for Host Applications using a Windows-initiated process (WIP). A WIP transaction is a host transaction that is originally initiated from the Windows platform. WIP is implemented via distributed program calls on the AS/400.

The first step in exposing the host-based transaction is to create a Transaction Integrator project in Visual Studio. Once the project is created, a .NET Client assembly should be added to the project. This .NET Client assembly encapsulates the information needed to translate function and data between the Windows and host platforms. When the .NET Client assembly is added, a wizard is launched to help with configuration. The first step is to choose the interface name and the types that will be supported. Knowing that this transaction will be used by BizTalk, BizTalk Adapter for Host Applications was selected as the type restriction as shown below:

Figure 11 .NET Client Assembly Wizard
Bb507734.7278f899-6797-4e8d-8251-18094ecf7826(en-US,BTS.10).gif

The next step in the wizard is to configure information about protocol, target OS, and programming model. In this case, TCP/IP was chosen as the protocol, OS/400 was chosen as the target environment, and the Distributed Program Call (DPC) programming model was chosen. It is worth noting that these are the only protocol, target, and programming model available for the IBM AS/400. (For more information about different programming models on different IBM host platforms within Transaction Integrator, see the documentation for the BizTalk Adapters for Host Systems.) In the case of the DPC programming model, the program library must also be defined.

Now that the interface is defined, a method needs to be added for the GFTRDCI inquiry transaction. Simply right-click the .NET Client Assembly in Visual Studio and click Import >> Host Definition. The import RPG wizard will be launched. This wizard allows you to import the GFTRDCI program in its entirety and extract the parameter list defined in the program.

Figure 12 RPG Import Wizard
Bb507734.d3a43138-8d41-40c5-ab77-cf39527d26fa(en-US,BTS.10).gif

Based on the parameters, the wizard also allows the user to specify whether each parameter should be treated as an in, out, or in/out parameter. When the wizard is complete, the .NET Client assembly is ready for use:

Figure 13 Host Definition View
Bb507734.146a2c2c-6d98-4101-a0c9-dfba4b8c0e86(en-US,BTS.10).gif

In Visual Studio, the .NET definition and the Host Data Definition (as shown above) are displayed. It also creates an XSD Definition that defines how it will be used with the BizTalk Host Application adapter as shown in the next figure.

Figure 14 Schema View
Bb507734.f95852f4-d563-4d7a-a333-b6c2539994b3(en-US,BTS.10).gif

This XSD can then be imported into the schema project and subsequently deployed to BizTalk Server. The Validate orchestration can be modified to use this transaction instead of calling the screen scraping services (SSValidate). The messages will be sent to a logical port called “TIPort” that will be bound to a physical send port using the HostApps adapter as defined below:

Figure 15 BizTalk Adapter For Host Applications Configuration
Bb507734.3a70e1aa-e7ab-4efc-8e77-c299c14baf75(en-US,BTS.10).gif

The HostApps adapter is one of the most flexible adapters used within Microsoft BizTalk Server. This adapter allows a single send port to handle multiple types of transactions. It also allows the transaction to use either the default remote environment or a specific remote environment. A remote environment abstracts connection information about the host system from the application itself. The remote environment can also be chosen on the fly from the orchestration by using BizTalk message context properties, which is helpful during service windows in which the main host environments are being maintained.

Before the solution will work, the .NET Client assembly must be deployed to at least one remote environment. The assembly can be deployed from Visual Studio directly, or by using the TI Manager tool that comes with the BizTalk Adapters for Host Systems.

After the successful implementation of the gift card service, the retailer decided to continue their SOA initiatives and implement their sales tax modules as a corporate sales tax service. In fact, instead of implementing the tax service themselves, they quickly outsourced this to a third-party service that maintained all of the rules around taxation in many different regions and countries. The contracts for accessing sales tax data were defined by the retailer and BizTalk Server was used to interact with the third-party sales tax service. This service could provide sales tax functionality for their brick and mortar stores right along with the e-commerce channels.

This brought up a unique challenge for the retailer. A large amount of order-processing logic that still resided on the host system needed to access the tax calculations. In the past, the only integration they had done from the host system to the Windows platform was through flat files and FTP. They needed something that could be executed in real time.

They accomplished this goal through the host-initiated processing (HIP) feature of Transaction Integrator. HIP allows a .NET assembly on the Windows platform to be configured and called by the host system in real time. As with the WIP transactions defined previously, the HIP transaction handles all of the data type conversions. The retailer built the .NET server assembly to call the sales tax service. In essence, this allowed the sales tax service to be called directly from the AS/400 and allowed the host system to become a consumer of the corporate tax service. HIP is a very powerful and important tool included with the BizTalk Adapters for Host Systems, and is beyond the scope of this document.

Whether exposing a legacy system, third-party product, or just a custom-developed application, using service-oriented techniques can provide some great benefits to the enterprise. First, the loose coupling provided allows the organization to modernize back-end systems while still servicing existing consumers. The retailer could decide to outsource their gift card system completely to another company, as they did the sales tax service. If the contracts remain the same, the upstream systems need not be totally aware of the changes.

Techniques like this can also allow enterprises to add long-overdue redundancy to their host systems. Using smart caching techniques in combination with the tools described previously can afford solutions that will be available even during host system maintenance windows. As functions are migrated completely off the host systems, enterprises typically see the benefits of the Windows platform and the high-availability features that come with it. With combinations of storage area networks and clustered servers, a highly scalable and reliable environment can be architected that is typically more cost effective than on the host system.

SOA also provides the capabilities to have existing systems interact with more modern business process management (BPM) and workflow tools, providing information workers and analysts the tools they need to remain competitive in their respective industries. As infrastructure platforms begin to catch up with the promise of SOA, it makes sense to take a low risk and incremental approach and to begin leveraging some of these advancements in technology.

Obviously, some of the topics described here are an oversimplification of some of the issues faced with SOA and interactions with host systems. Versioning, security, operational, and performance considerations have to be thought through completely before implementing a solution like this. Using a platform like Microsoft BizTalk Server 2006 helps ease the burden of some of these considerations by providing an operational platform that helps manage some of the decoupling of each of the core systems. Taking an opportunistic approach lets an organization handle each of these topics in a small, incremental fashion and lowers the organization’s total risk while providing the key benefits of a service-oriented architecture. Organizations can leave existing business rules where they are and migrate them to new platforms as appropriate by applying these concepts to their host systems. This allows these systems to be accessed by upstream systems that usually had no means to access the important rules and data in the near term, and provides greater flexibility and agility as the organization grows in the long term.

About the Authors

Chris Kabat and Bill Topel are both principals for MPS Partners, a Chicago-based services firm that focuses on turning vision into value through people, empowered by technology.

Chris serves as a principal architect and is the Director of MPS Partners’ Connected Business Systems service line. He is also a member of the Microsoft BizTalk Server Virtual Technology specialist team. Chris has been working with the integration of IBM host systems for over 10 years, starting his career as a COBOL programmer.

Bill is the managing partner of MPS Partners and has worked with host systems for over 25 years. He has authored over 35 articles on Microsoft and IBM interoperability and is a frequent speaker at local and regional user groups.

--------------------------------------------------------------------

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2007 Microsoft Corporation. All rights reserved.

Microsoft, MS-DOS, Windows, Windows Server, Windows Vista, BizTalk, and Visual Studio are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

All other trademarks are property of their respective owners.

Show: