Mark Berman
Microsoft Corporation
Peter Kelly
3Sharp, LLC
August 2005
Applies to:
Enterprise Architecture
Solution Architecture
Service Oriented Architecture (SOA)
Application Integration
Summary: This case study presents the benefits of adopting an SOA approach for dramatic improvement to operations. Host Integration Server 2004 exposes server functionality to Web services, making it easy to expose disparate LOB systems, and BizTalk Server 2004 makes it easy to create complex business logic that is centralized, simple to manage, and easy to integrate with front-end applications. This service-oriented solution exposes previously unreported error information via Microsoft Operations Manager, making the entire infrastructure easier to manage. (11 printed pages)
Click here to download the Enterprise SOA Application Demo.
Contents
Executive Overview
Introduction
Challenges
The Service-Oriented Solution
Benefits of the Service-Oriented Solution
Conclusion
Additional Resources
Executive Overview
In the past few years, there has been a lot of discussion about service-oriented architecture (SOA) and the benefits it can bring to organizations, especially those with disparate line-of-business (LOB) systems. As a sign of the maturity and viability of the SOA approach to enterprise application integration, this year saw the release of the broadly-endorsed SOA Blueprints by Middleware Research (available at http://www.soacenter.com). ISVs, consulting firms, and customers alike are beginning to speak the same language, and adoption of service-oriented solutions is dramatically increasing. Computerworld recently reported that:
"2005 will be the year of the SOA… According to The Yankee Group, 75% of firms plan to invest in the technology and staffing to enable a service-oriented architecture (SOA)." —Computerworld November 15, 2004
To illustrate the value of SOA, this paper describes the steps taken by Microsoft to meet some major application integration challenges it was facing in its Microsoft Technology Centers, challenges similar to those faced by most enterprises today. The promise of SOA will become clear as we explore these challenges, their root causes, and the way in which the service-oriented solution dramatically improved operational efficiencies and the ability to flexibly interoperate with a variety of LOB systems.
This paper will also show how critical functions within the service-oriented solution were served by Microsoft technologies. For example, Microsoft Host Integration Server 2004 made it possible to expose IBM Midrange-based and Mainframe-based applications as Microsoft .NET-based Web services with all functions fully available to Windows applications. This enables much greater interoperability and also provides a robust solution for error reporting and error handling. Using Microsoft BizTalk Server 2004 to manage the orchestration of business processes, the Microsoft Technology Centers were able to take control of business processes, by, for example, checking for bad data before submitting it to an application. In the detailed description below, you will see other elements of the solution, both generic and Microsoft-based.
While Microsoft firmly believes that choosing to implement a service-oriented architecture is a business decision and not a technology decision, it is still true that the technology you use matters. This paper will show that, in terms of ease of development and ease of integration and management, Microsoft technologies are well-suited to help you address the business challenges you face today. For example, using Microsoft development and middle-tier technologies, the entire service-oriented solution described below was built in about two man-months.
Introduction
Microsoft Technology Centers (MTCs) provide customers with an environment to envision, architect, and then evaluate a solution using Microsoft and partner technologies prior to deployment in their own IT environments. While the "business" of the MTCs is to facilitate these offerings, it and its 13 connected locations around the world operate like any other enterprise, with all the concerns of performance, security, reliability, manageability, and so on.
The global MTC infrastructure is designed like an actual enterprise environment. Like many companies, the major challenges faced by MTCs revolved around the need to expose existing legacy applications not developed using Microsoft technology and not running on a Windows platform to front-end applications across the environment. Specifically, each regional location had its own infrastructure and accessed a group of LOB applications running at the main office (that is, in Redmond). These shared resources are used for key business processes by front-end applications both at the main office and at the regional offices.
As the number of transactions from the various locations increased, the requirement to provide reliable shared access to the main office shared resources became increasingly difficult to meet. For example, the applications running on the Midrange (Report Program Generator or RPG) and the Mainframe (COBOL) systems are older applications that were not designed to handle multi-access. As a result, regional offices running applications at the same time would encounter locking issues or intermittent failures, with no indication as to the specific cause or nature of the problem. The main office had a difficult time managing these applications and fixing problems as they arose, especially since the original developers of the applications were no longer available.
Rather than rebuild all of the MTC demo environment's LOB systems, the management team decided to review the existing integration requirements and refine the integration layer between the LOB and the front-end applications so as to:
- Improve stability by allowing multiple front-end applications to access LOB resources simultaneously and by having business processes that cleanly handle failed transactions.
- Decrease troubleshooting time by providing actionable error information.
- Enable health monitoring to prevent problems from occurring.
- Define consistent contract information.
- Define consistent binding information.
Challenges
Taken together, the MTCs operate with similar systems and challenges as a typical enterprise. To better understand the specific problems the MTCs face, we will discuss the topology.
A Global Infrastructure
The MTC infrastructure represents a fictional mid-sized manufacturer of retail goods that receives and fulfills orders. The regional offices each have the required infrastructure (domain controllers, messaging servers, Web servers, SharePoint infrastructure, and so on) to initiate and interact with an order. As shown in Figure 1, the Main Office has the Midrange (iSeries) AS/400 server (which hosts the RPG based Warehouse application), the Mainframe zSeries 890 server (which hosts the Custom Information Control System [CICS] based Shipping application), and the RS/6000 UNIX server (which hosts the WebSphere-based Fulfillment application), as well as a home-grown call-back application running on Windows.
.gif)
Figure 1. Global Infrastructure
Based on this infrastructure, each regional office accesses the shared resources to demonstrate a variety of realistic business processes. A key business process involves a regional office initiating an order using a Web application or a Microsoft Office InfoPath 2003 form. This action initiates a complex business process that uses all of the LOB systems. The basic business process is as follows:
- An order from a regional office is sent to a shared Warehouse application running on the Midrange system.
- The Warehouse user uses an IBM 5250 emulator to view the order and update the order status from New Order to Allocated.
- A smart device picker application receives a Pick request.
- A Warehouse worker performing the Pick action acknowledges that the item has been picked.
- Order status information is updated to Picked on the Warehouse application.
- A Warehouse worker updates the order status from Picked to Shipped.
- The Mainframe Shipping application receives the Ship request, which can be verified using an IBM 3270 emulator.
- The WebSphere Fulfillment application receives the Fulfillment request.
- The Call Back server notifies the purchaser that the item will be shipped in an agreed period of time.
Rigid, Unmanageable Applications
Before applying service-orientated principles, the system had the same manageability issues of many applications that expose legacy systems. Primarily, the LOB applications could fail during simultaneous access attempts from front-end applications, from network outages occurring mid-transaction, or from badly-formatted data. When a request to an application did fail, the applications did not expose error information, and they did not cleanly shut down or restart (where appropriate). From an operations perspective, troubleshooting without proper error information was a challenge.
Additionally, because of the tightly coupled business logic and business process, the architecture could not take advantage of new technologies. This was particularly problematic for the MTCs, since their business is to showcase an end-to-end better-connected story that highlights the best in new technologies and practices.
In the end, the LOB applications were rigid and unmanageable, and the operations team was in constant reaction mode, trying to keep these shared resources working but without any true sense of confidence that there would not be a failure on any given day.
Solution Vision
The MTC team began with a belief that a service-oriented architecture could help expose the various LOB applications to facilitate more flexible integration, improve stability, and provide meaningful error information that would assist in troubleshooting. With this proposition, they established a set of constraints and requirements that any proposed infrastructure would have to meet.
The overriding goal was to integrate with existing Mainframe and Midrange applications, and have more efficient interactions with the applications while maintaining flexibility, interoperability, and a standards-based approach to instrumentation.
The constraints that shaped the service-oriented solution were:
- All public interfaces must be exposed as Web services so that:
- All integration is implemented in a consistent manner without dependencies on local assemblies.
- External developers can independently integrate with the solution through well-defined access points (bindings).
- External developers can independently integrate with the solution through a well-defined, consistent schema (contract).
- Detailed external application and infrastructure (for example network) error information must be made available in a consistent and predictable way.
- To facilitate interactions and troubleshooting, there must be an end-to-end reliable-messaging delivery (where appropriate)
- Due to the frequency of changes on the client tier, there must be support for loosely-coupled client integration.
The Service-Oriented Solution
Figure 2 shows the service-oriented solution for the main office shared resources.
.gif)
Figure 2. The SOA solution for the Redmond facility shared resources
All elements in the shaded green area represent new or altered components. Note that the call back server, which was a home-grown solution, was replaced with Microsoft Speech Server, a tool for deploying and managing distributed speech applications. In brief, the service-oriented solution abstracts all shared resources behind a series of Web Services and BizTalk Orchestrations. All front-end applications interface with this abstraction layer.
A front-end application, such as the order form described above, initiates a transaction by sending an XML-formatted order to the main BizTalk orchestration. The orchestration evaluates the order and executes a business process whereby the new order details are sent to other Web services that interact with the various LOB applications.
The key is that the service-oriented solution, and only the service-oriented solution, directly interfaces with the LOB applications. As a result, the Web services can reject badly formatted orders before they reach the LOB applications, solving a common problem the MTC experienced before. Also, because the business process orchestration provided by BizTalk Server 2004 is asynchronous, multiple orders can arrive and be handled together without locking up the LOB applications.
Finally, when the far less frequent errors and exceptions do occur, whether at the LOB application layer, the network layer, the host integration layer, or the orchestration layer, rich information about the event is propagated through the Web services to the Operations team, where it is actionable and available for analysis using Microsoft Operations Manager (MOM).
SOA Components
The SOA components are a series of interconnected layers. From a developer's perspective, talking about the layers also represents the logical approach to building the service-oriented solution. Thus, the first task is to make it possible to access the LOB applications through Web services. Following that, it is necessary to implement business processes in some way.
Making Back-End Applications Available Through Web Services
Because the Warehouse application was written in RPG and the CICS application was written in COBOL, they could not natively be accessed through Web services. Using Host Integration Server 2004 (HIS), the underlying application interface can be exposed as a .NET-callable Web service, where application functions are now available directly within Visual Studio .NET (the Host Integration Server is shown in Figure 2).
The WebSphere application was exposed through a Web service by creating a .NET proxy object using built-in .NET Framework tools.
For the Speech Server, the team decided that it actually made sense to simply replace the existing solution with Microsoft Speech Server, which offered a much more robust solution out of the box. Some interesting development techniques were used to produce sophisticated Speech functionality. Speech Server dynamically generates an appropriate message based on pre-defined WAV files that contain generic representations for things such as integers used in order numbers and common welcome messages. Creating the Web service to expose this server's functionality was a simple matter, since Speech Server supports .NET natively.
With the Web services in place, the essential elements of application abstraction were accomplished. All event and error codes were captured and directed to MOM, where they can be monitored and analyzed to diagnose and prevent problems.
Using Business Processes to Take Control of the Infrastructure
Beyond simply abstracting LOB applications, a good service-orientated solution also provides a flexible means of controlling business logic. Using BizTalk Server 2004, orchestrations were used to expose multiple Web service facades as a well-defined business process. In essence, a single orchestration acts as the interface for all front-end applications transacting with a specific LOB application.
This orchestration acts as the director of the orchestra, so to speak, determining when and how every transaction is processed. In the business process example given above, where an XML-formatted order form is submitted, the orchestration evaluates the order and submits it to another orchestration, which is designed to manage interaction with the Warehouse application. Other orchestrations manage interaction with each of the other respective LOB applications.
Note While Figure 2 only shows the BizTalk Server residing above the LOB Web services, there are, as just described, multiple orchestrations that manage this particular business process.
These orchestrations enforce orderly access to LOB applications, and they also prevent many of the kinds of problems that would have caused a failure in the past. For example, the orchestrations will not allow badly-formatted orders to proceed, and will write the failure and cause (if known) to the event log.
Another benefit of using BizTalk Server 2004 is that it allows the real-time visualization of the status of an order within the system. For example, an order may make it to the Warehouse application, where the Warehouse operator is supposed to change the order's status to "Pick", indicating that the product is in stock and that the order is ready for shipping. Should some failure occur, operations personnel are able to view the live running instance, confirm that the order is still sitting in the Warehouse application awaiting status change, or further diagnose the cause for delay or failure.
Getting Proactive
As mentioned above, the management team realized that, because they were exposing error information in a consistent way, they had an opportunity to use this information not only for troubleshooting and responding to problems but also for proactive system monitoring. They are using MOM to collect all error and event information from the Web services, and they are able to perform diagnostic analysis using this information.
Building the SOA
According to Computerworld's recent report:
"SOA doesn't require big-bang investments in technology with nebulous ROI and prolonged payback periods. SOA is achievable in small increments, and the tangible benefits of software and services reuse immediately deliver bottom-line results. In fact, there is often a clear ROI at the project level while an organization builds toward the strategic vision of an SOA. This phenomenon is one we're unaccustomed to, and often business executives are skeptical. However, the proof is in the pudding." – Computerworld November 15, 2004.
The MTC case study certainly corroborates this statement. The entire service-oriented solution, including the HIS interface for the RPG and COBOL applications, all the Web services, the BizTalk orchestrations, the new Speech Server and all its logic, and the new InfoPath-based smart client application for purchasing, all took about two man-months to build. Of course, much of this is due to the ease of use for HIS, BizTalk, and the .NET development platform, but the reality is that a small investment in up-front design helps reduce the overall development time, and as a result has an enormous effect in terms of being able to take control of legacy systems. Also, as the Computerworld quote points out, once this investment is made, similar projects are that much easier to do, making SOA possible on a project-by-project level, rather than as an enterprise undertaking.
Benefits of the Service-Oriented Solution
The service-oriented solution has a number of benefits, including:
- The layer of business logic between the users and the legacy systems ensures that the systems are being accessed in only the ways they were originally designed.
- The legacy applications are now available (through HIS) as .NET applications, making development a much more rapid process.
- BizTalk's routing support centralizes the entry point to the myriad of LOB systems. BizTalk simply analyzes the XML-structured data information it is given and decides where it needs to go. Also, modifying business logic does not require completely rewriting the application. Rather, a developer simply needs to update an orchestration and redeploy it.
- Because the legacy systems are now easy to access from the .NET platform, extensibility is easy. Applications from Microsoft ASP.NET, Office System, and Smart Phones all have the potential for interacting with data that resides within the LOB systems.
- Developers do not have to specialize for each legacy system. Each system is accessed through a set of common interfaces.
- The service-oriented solution keeps track of errors at all levels of the process except for the originating client. This is as it should be. Developers interacting with the service-oriented solution are responsible for the piece of the application they write (the client). From BizTalk to the Web services to the legacy applications themselves, the operations staff is notified if an error occurs. Often times, the problem is resolved before any of the external clients realize there is one!
Conclusion
This case study demonstrates the benefits of adopting an SOA approach to dramatically improve operations. HIS exposes server functionality to Web services, making it easy to expose disparate LOB systems, and BizTalk Server 2004 makes it easy to create complex business logic that is centralized, easy to manage, and easy to integrate with front-end applications. By exposing previously unreported error information by means of MOM, the service-oriented solution makes the entire infrastructure easier to manage.
The details of this particular service-oriented solution are unique to the particular needs of the MTCs. However, the basic principles of abstraction through the use of Web services and orchestrations represent the best thinking on how to achieve an infrastructure that costs less, is more flexible, reusable, and leverages existing investments in technology.
Additional Resources
Product Sites
SOA related sites