Hybrid Reference Implementation Using BizTalk Server, Windows Azure, Service Bus, and Windows Azure SQL Database
Author: Valery Mizonov
Integrating an on-premises process with processes running in Windows Azure opens up a wide range of opportunities that enable customers to extend their on-premises solutions into the Cloud environment.
Based on real-world customer projects led by the Business Platform Division’s Customer Advisory Team (CAT), this reference implementation comprises a production quality, fully documented hybrid solution that demonstrates how customers can extend their existing on-premises BizTalk Server infrastructure into the Cloud. The solution is centered on the common requirements for processing large volumes of transactions that originate from the on-premises system and are then off-loaded into Windows Azure to take advantage of elasticity and on-demand computing power of the Cloud platform. The reference implementation addresses these requirements and provides an end-to-end technical solution that is architected and built for scale-out.
The main technologies and capabilities covered by the reference implementation include: Windows Azure platform services (compute, storage), Windows Azure Service Bus, Windows Azure SQL Database, and BizTalk Server 2010.
The reference implementation is founded on reusable building blocks and durable patterns widely recognized as “best practices” in Windows Azure solution development.
This project is implemented as a hybrid solution in which BizTalk Server represents a fundamental dependency. Logically, BizTalk Server customers can install and use the reference implementation with a minimum of modifications. However, the solution also carries many of the reusable patterns and building blocks that developers can explore in isolation from the larger end-to-end reference implementation.
The accompanying source code is available for download from the MSDN Code Gallery.
This section describes the components used in this solution.
The BizTalk Server 2010 is a core element in the hybrid solution architecture. The transactions that need to be processed in the Cloud originate from a BizTalk application running on the premises. The BizTalk application also hosts the service endpoints, which make mission-critical capabilities such as data transformation, BAM and BRE available to the cloud-based applications.
Business Activity Monitoring (BAM)
The BAM is used to store activities that are generated and tracked by the BizTalk Sever application and cloud-based services. All activities are collected into an on-premises BAM database and are made visible through the BAM portal.
Business Rules Engine (BRE)
The BRE is used by both the on-premises BizTalk application and cloud-based services to power decision making and define the operational aspects of the data processing on the Cloud. In addition, the BRE authorizes and manages complex application configurations used by the hybrid solution. Lastly, the BRE policies provide the extensibility mechanism upon which custom activities can occur at runtime depending on type or content of messages flowing from and to the BizTalk application.
The BizTalk Mapper tool authors complex XML transformation rules and packages them into maps. The BizTalk maps are then executed on the Cloud to transform XML documents between canonical and custom schemas.
Windows Azure Service Bus
The Service Bus provides the endpoint in the Cloud through which all traffic between the on-premises and cloud-based services (both directions) is securely relayed. The Service Bus is also leveraged to enable the exchange of event notifications between worker roles running on the Windows Azure platform.
Windows Azure Worker Role
The worker role processes messages by pulling messages from Windows Azure queues and shredding the batches of business transactions into individual records that are stored in a SQL Database.
Windows Azure Storage
The storage queues are used to hold references to messages that are in line to be processed. Because the size of each item in the queue is limited, the actual messages are stored temporarily in blob storage.
Windows Azure SQL Database
The SQL Database provides durable persistence of the business transactions to the Cloud. The large transaction sets are stored in a SQL Database, from which they are debatched into smaller XML documents and dispatched for processing.
Hybrid Solution Architecture
The following diagram depicts the architecture of the hybrid reference implementation:
Best Practices Applied
The following best practices were applied in the technical design and implementation of the hybrid solution:
Reliable WCF communication using WCF clients and WCF hosts capable of recovering from transient communication errors.
Windows Azure storage abstraction which provides a generics-based enhancement for the Windows Azure Queue Storage API that can store very large messages (to overcome the current message size limit).
Transient Fault Handling Framework, which delivers a reliable SQL Database data access layer that can detect and recover from transient conditions such as SQL Database throttling.
Centralized configuration store capable of describing very rich configurations and storing them securely on-premises.
Inter-role communication using event relay messaging provided by Windows Azure Service Bus.
Hybrid instrumentation framework for high-performance real-time logging and tracing of cloud-based and on-premises processes into an ETW event collector.
WCF custom endpoint behavior for securing and simplifying the integration of BizTalk Server and Windows Azure Service Bus.
Windows Azure queue listener that can auto-scale up and down and reduce overall storage transaction costs.
For more information, please refer to the resources linked to each of the individual best practices.