Enterprise Integration: A Practical Guide for the Retail Solution Architect
Nick Lewis, Industry Technology Strategist
Microsoft BizTalk Server 2006
Summary: Serving the needs of both business stakeholders and enterprise architects requires a practical approach to enterprise integration and service oriented design; learn how with tips in this article. (8 printed pages)
Admittedly, enterprise integration is a fairly treadworn topic; however, as retail solution architects, we're challenged by competing priorities. In an industry driven by operational efficiency and cost containment, software development projects must demonstrate compelling ROI. In other words, architects must design solutions that solve current and future business needs as cheaply as possible. At the same time, software must promote reuse, scale to meet unplanned demand, support enterprise security standards, and leverage enterprise-provided services. In other words, solution architects have implicit responsibility to build software that plays well in the enterprise. Serving the needs of both business stakeholders and enterprise architects requires a practical approach to enterprise integration and service oriented design.
So why are solution architects concerned with enterprise application integration (EAI) and service orientation? Well, it turns out that these subjects aren't merely in vogue right now; they have practical implications. The most obvious is that solution architects should use existing, loosely coupled services available in the enterprise to address application requirements. However, there's a more important reason you should care about enterprise architecture. The two topics, solution architecture and enterprise architecture, are tied closely together through project requirements, organizational initiatives, and products that help us effectively build both. For instance, products like Microsoft BizTalk Server 2006 solve EAI challenges. However, these platforms, broadly categorized as business process management (BPM) servers, help solve many conceptually smaller solution architecture challenges. So, it's very likely achieving your organization's service oriented architecture (SOA) objectives will leverage services you've built or bought to solve smaller project needs.
Interestingly, as a solution architect, you're in a key position to influence whether your organization actually buys and uses these technologies. Solution architects, especially those supporting lines of business, tend to work on projects with "legs." These projects are initiatives that generally are budgeted, have concrete deliverables, solve a particular business problem, and have direct benefit for business stakeholder. So, components of an organization's enterprise architecture are often those acquired to support various business solutions. Unfortunately, many of these are acquired with little thought as to how they might interoperate to achieve common service goals. Because business solutions drive the acquisition of tools and code artifacts, they provide the fabric of the enterprise architecture.
Of course, enterprise architects are increasingly important in retail organizations; they develop, communicate, and enforce a vision that leverages technology investments across all solutions and components. They also ensure that solutions support the organization's approach to service orientation and software reuse. However, implementing an envisioned SOA is a highly iterative process. No company in today's retail industry can afford to rebuild software and hardware investments just to provide the benefits of service orientation. So, enterprise architects often work with existing software investments and try to influence future software acquisitions and development activities.
So, how does building an SOA infrastructure benefit solution architects directly? Obviously, there are long-term advantages to having a collection of loosely coupled services available for use. For instance, you get to take advantage of these services while making them available for the next project down the road. However, designing for service orientation isn't free. Minimally, it requires more cross-team collaboration, review, and testing than "traditional" services. Business stakeholders, on the other hand, are generally focused on building point solutions to solve specific problems. Even though most organizations understand the long-term software cost and speed-to-market benefits offered by implementing SOA pricinples, the time and cost impact on a single project could be overwhelming. Accordingly, solution architects often have difficulty getting business stakeholders to invest in building SOA-friendly solutions.
So, by now you might be asking yourself:
- I've read technical case studies like the one for Virgin Megastore which shows an example of using BizTalk Server to provide fraud detection services. But that's just one service. What does a retail industry-specific SOA look like?
- If SOA is achieved iteratively over time, what are some opportunities that I should be looking for as I'm designing solutions and services?
- I see how BizTalk can be used as a component for providing service orientation, but it's way too big and complicated for the problem I'm trying to solve. How am I supposed to get started acquiring these enterprise components?
Let's address the first question. "What does a retail-specific SOA look like?" As a solution architect, you could map the problem you're attempting to solve with an ideal, enterprise architect driven view of your retail enterprise. Of course, this view changes significantly for specific retail industry segments and business sizes. Also, most organizations simply don't have a 5-10 year roadmap for SOA. So, you might then ask, "which components of my application architecture should I provide as loosely coupled, message-oriented services, and which should I build and couple tightly to my solution?" For instance, you might expose a Price Lookup (PLU) operation as a core service that exhibits SOA behaviors. However, you might tightly couple a data caching service that maintains PLU information for offline operation. While the PLU operation might provide a useful service for many applications, the data caching behavior may only be useful for a single scenario on a point of sale terminal. Figure 1 illustrates service orientation concepts:1 (Click on the image for a larger picture)
The diagram describes a model of a composite application composed of a set of services that exhibit particular behaviors. Specifically, the services manage state; exchange messages; are bound by contracts; and are governed by various policies. That's a mouthful, and it seems like a lot of extra overhead. However, these are necessary for a service designed for easy reuse, versioning, and scalability. However, not all application behaviors are good candidates for service orientation.
So, exactly which behaviors are good candidates? The truth is that they vary wildly from organization to organization depending on aggregated business needs. Variability between various retail business models makes it difficult to provide an effective, one-size-fits-all SOA template. Consider the differences between speciality and discount retailers. Although they share a number of common business practices, point of sale, logistics support, and human resource management, they do so with varying degrees of rigor and focus. For instance, discount retailers generally require more redundant and scalable software systems to handle large numbers of transactions and longer hours of operations than do specialty retailers.
When deciding whether an application behavior should designed for service orientation, solution architects should consider the following:
- Is this operation relevant if I separate it from my particular need?
- Can I create an explicit boundary around the operation so that it can operate independently?
- Can I describe a message format for communicating with this operation that is generally self describing and reusable?
- Can this operation manage its own state?
- Will adding additional service-oriented features to this operation provide sufficient future value to justify the time and effort required?
If you evaluate your application operations and identify those that match these criteria, you'll produce a list of candidates to design and construct as loosely coupled services. From a candidate list, you should enlist feedback from other stakeholders, especially those interested in enterprise architecture. This is one of the more important communication flows if you're serious about designing for service orientation. As a solution architect, you should be somewhat narrowly focused on stakeholder needs. Your job is to rationalize the behaviors of your application and offer up candidates for enterprise reuse. Keeping this mission in mind makes it easier to distinguish between solution and enterprise architect responsibilities.
Let's now look more closely at the second question. "If SOA is achieved iteratively, what are some opportunities that I should be looking for as I'm designing solutions?" This is where a solution architect can, in my opinion, most effectively contribute towards evolving enterprise architectures. Setting SOA aside for a minute, a good solution architect should seek to avoid building new services if they're already available. We make design choices to shorten the development life cycle as much as possible while maintaining quality and meeting business needs. That, in turn, shortens the time-to-value delay for reaping business value from the produced solution. As an example, let's look at the scenario described in Figure 2.
Figure 2. General inventory management scenario
In very general terms, Figure 2 shows an inventory management solution that ties together several loosely coupled systems using a BPM product, BizTalk Server. Let's consider how you might design this application without the assistance of a BPM server. First, you'd probably use an asynchronous queuing mechanism to pass messages between these systems. Then you'd have to design and construct components to wait on those queues, push messages to the next queue, or execute something when a message appears. This problem gets exponentially harder when you need concurrent access, high throughput, and durable message queues.
As a solution architect concerned about time-to-value and quality, you must look for existing software you could buy or extend more cheaply than you could build yourself. Assuming you have sufficiently complex business and quality requirements, BPM servers like BizTalk Server can address this business scenario very cost effectively. So, from the humble beginnings of an inventory management application, you might acquire an asset that can serve as a key component for providing enterprise service orientation. This is a classic way for SOA to take root in a large enterprise.
Figure 3 shows how this solution architecture might evolve and leverage BizTalk Server to exhibit service-oriented behaviors.
Figure 3. Aggregating services3
The same BPM orchestration driving the inventory management application can be reused as required behaviors are needed. The new service endpoint can expose business operations of the inventory management solution in a service-oriented way. Although the inventory management solution architecture doesn't require it, you get built-in support for exposing the new service when using BPM servers such as BizTalk Server. Of course, BPM servers don't relieve the solution and enterprise architect from designing the service endpoint so it exhibits SOA behaviors described in Figure 1, but they do provide a way to quickly implement it. Further, existing schemas from standards organizations such as OTA and IXRetail help solution architects design service endpoints that address the SOA imperative for interoperability.
This, I believe, is a more realistic way of considering and evolving service orientation in the enterprise. Frankly, it's rare for organizations have the foresight to determine which business operations should be exposed as enterprise services. Using tools like BizTalk Server, solution architects have a way to address the need as software services are required across and outside the enterprise.
Most organizations conduct software reviews. Typically, these start at the beginning of the design requirements phase and continue through project completion. Generally, these reviews identify opportunities for reuse, uncover or highlight security considerations, and determine design fit with business requirements. During these sessions, you might also look for design requirements easily addressed with BPM servers2 such as BizTalk Server. These requirements include:
- Communication services that allow interaction with various business services needed by your application.
- Complex operations that require services such as state management and transaction support.
- Tools for defining the logic that drives your application.
- The need to map between data formats used by various business services.
- Allowing the business rules an application uses to be created and managed separately from the process flow.
- Allowing people to participate in the business process this application implements (this is broadly known as human workflow).
- The need to provide process monitoring services. This might include both monitoring at a technical level and business activity monitoring (BAM) that provides real-time information about a running, distributed process applying these data maps and business rules.
This should give you some criteria for identifying opportunities to build point solutions that leverage BPM servers. Just as you consider grouping functionality into classes and services, you can partition application behaviors to identify which pieces could be solved in a cost-effective way by enterprise BPM services.
Finally, let's look at the third question. "Okay, I see how BizTalk Server can be used as a key component for service orientation, but it's way too big and complicated for the problem I'm trying to solve. How am I supposed to get started acquiring these types of components?" This is one of the biggest challenges a solution architect faces when considering build versus buy: return on investment (ROI). Solution architects frequently have to make design choices that are evaluated with a very limited scope. In a best case scenario, you might propose designs that demonstrate strong ROI across a number of related projects. But, frequently you have to show positive ROI for a single, small project. Since BPM servers are enterprise resources that provide a number of services, it's difficult to introduce them as part of solution architectures except in the largest projects.
This is a type of "Tragedy of the Commons" problem where common services will benefit the organization but not individual stakeholders. Naturally, this produces several negative impacts for retailers. The biggest might be you end up developing and supporting multiple software architectures to solve the same kinds of issues. For instance, you might use a homegrown messaging and orchestration solution for communicating with a small supply chain partner because the budget for that project is very small. However, later on, you might build a more robust, BPM-based, trading solution for communication with a larger partner. Ultimately, you have two or more solutions for supply chain communication that have to be maintained or, worse yet, integrated.
There's no easy answer to this problem, and it requires the solution architect to enlist the support of others. This is a great place to seek sponsorship from those supporting the overall enterprise architecture in your company. Many organizations will supplement project budgets with infrastructure funding when there are good business cases for solving a small problem with components that show long-term value. But, let's face it; this can be a very tough road. This process can extend project timelines, involve lots of additional stakeholders, and generally fall into a political quagmire. This is one reason that organizations tend to overbuild homegrown solutions. Obtaining consensus for acquiring enterprise components and services is notoriously difficult. I recommend keeping your eye on the business stakeholder first and foremost. By evaluating requirements and offering design alternatives that demonstrate vision and provide long-term ROI, you've done your job.
Solution and/or application architects are in a unique position to influence enterprise architecture. Although we might not have a specific responsibility to foster service orientation across the enterprise, we frequently make application design and construction choices that have a great impact on those capabilities. Collectively, we build or buy components that enterprise architects seek to organize, manage, and leverage across and outside the organization. That doesn't mean that we need to take our eye off our primary mission to design great software that solves business problems. The goal of a BPM server such as BizTalk Server is to make it easier to create, run, and manage the process logic that drives a composite application2. That is valuable behavior, but it should stand on its own and provide good ROI for a particular business need. However, introducing these components into the enterprise provides great long-term benefits. So solution architects should evaluate requirements and design choices to support SOA goals. This is a great way to organically grow an SOA while still being true to your business customers.
1 Michael C. Woods, Microsoft Corporation, July 2004
2 David Chappel, Understanding BPM Servers, October 2004
3 David Chappel, BizTalk Server in a Service-Oriented World, June 2004