Export (0) Print
Expand All

Improving Ordered Message Delivery in BizTalk Server 2006 R2 During Concurrent Processing

Microsoft Corporation

Applies to: BizTalk Server 2006 R2

Published: September 2007

Author: Bernard Pham

Contributors: Jean-Emile Elien, Lee Graber, Kartik Paramasivam, Jayu Kati, Jamie Broomall, Kevin Lam, Stuart Landrum

Summary: This white paper describes a solution for improving the ordered delivery of messages in BizTalk Server 2006 R2 during concurrent processing by BizTalk orchestrations when used to implement business processes.

Download Location for Solution: To download the solution that is referred to this in whitepaper, click here.

Ordered delivery of messages in Microsoft® BizTalk® Server 2006 R2 is determined when a message is persisted into the MessageBox database. The end-to-end requirements for ordered delivery of messages between receive and send ports consist of using a serialized receive adapter (MSMQ, MSMQT), and marking the send port “Ordered Delivery” in BizTalk Server 2006 R2. This works in a pure messaging scenario where there are no orchestrations involved, although it is not sufficient to guarantee order when multiple orchestrations are in use. This white paper describes a solution that improves the ordered delivery of messages in complex messaging scenarios involving multiple orchestrations.

Figure 1 describes a typical example of ordered delivery in BizTalk Server. It represents a data processing system for a hospital patient, where messages that pertain to the patient’s case arrive at the receive port in sequential order: M1, M2, and M3. Using a serialized receive adapter ensures that reception order is maintained all the way into the MessageBox database.

Figure 1. Ticket Dispenser and Gatekeeper Ordered Delivery Solution
Bb851740.1542441d-d33a-4634-898f-c89efb0d94fa(en-US,BTS.10).gif

Business deployments incorporate orchestrations to model different parts of a business process. For example, an orchestration to admit a patient, an orchestration to handle updates of patient records due to blood work or an x-ray, and an orchestration to bill the patient. Each orchestration subscribes to the appropriate message control ID representing different message types. The MessageBox database publishes messages accordingly and fulfills the appropriate subscriptions to the orchestrations.

To increase system throughput, businesses run orchestrations in parallel to process messages concurrently. Messages going through parallel processing chains can compromise message order, because messages are of different payload sizes, and different types of messages may take a different amount of time to process, which introduces latencies into message processing. For example, it takes longer to process a message to admit a patient, because different databases in different parts of the system need to be updated, than it does to process a message to update a specific record for that patient.

Even though the MessageBox database publishes the messages to the orchestrations in the order that the messages are persisted into the database, the order of messages sent out of the send port depends on the processing latencies of the business orchestration. (A business orchestration is one that implements a business process.) Even with the Ordered Delivery setting in the send port set, the overall goal of sending messages out in the exact order they arrive at the receive port cannot always be achieved. It is important to note that this is not a limitation of BizTalk Server in handling message order. This business pattern requires additional custom components, which are described in this solution, to achieve end-to-end message order as defined by the business.

This white paper will show implementations that will improve end-to-end message order delivery between a pair of receive and send ports while running concurrent orchestrations. The architecture implements a common resequencer pattern to solve this problem.

The solution introduces the notion of a ticket for every message traveling through the processing chain. Business orchestrations can take the necessary time to process the messages in any order. As long as the ticket is propagated in each message, the messages can be resequenced on the way out.

It is important to define the boundaries of ordered delivery. For this solution, order begins at the BizTalk Server receive port, and the goal is to maintain this order until the messages are sent out of BizTalk Server.

It is likely that BizTalk Server is only one of many software components in, for example, a complex health care environment or a financial deployment. Upstream from BizTalk Server may be one or multiple mainframe computers generating messages with their own specifications for order, independent of BizTalk Server. Furthermore, downstream systems that consume ordered messages from BizTalk Server may have their own specifications for order. The point is that message order can vary depending on the system deployed for different business processes. In general, systemwide message order needs to be considered and designed in the system architecture, and it is not the responsibility of any one component like BizTalk Server, unless BizTalk Server is designed to be a message coordinator for the whole system.

Assumptions of the Solution

The following assumptions are made for this solution:

  • Message order is defined at the receive ports.
  • Messages are sent to BizTalk Server in sequential order by the upstream system.

Goals

This solution has the following goals:

  • Maintain message order between a pair of receive and send locations even when business orchestrations process the messages in parallel.
  • Maintain order across computer and server shutdowns.
  • Achieve maximum sustainable throughput performance of 10 messages per second.

Non-Goals

The solution does not consider:

  • Messages that are sent in order by the upstream system but arrive out of order due to network latency, which are beyond the scope of this white paper.

Requirements

The solution has the following requirements:

  • A receive adapter that supports ordered reception of messages.
  • Business orchestrations that have to propagate message ticket context properties and initialize gatekeeper correlation ID.

The design uses a custom receive pipeline component to stamp every incoming message with a sequence number. In addition, a singleton (single instance) orchestration resequences messages on the way out. The sequence number must be durable in order to maintain order across shutdowns. The custom pipeline component uses a SQL table to get the next available monotonically increasing sequence number and stamps the sequence number into the global context property for the message. The input messages to the pipeline component are assumed to be ordered using a receive adapter supporting ordered delivery such as MSMQ, MQSeries, MSMQT, and MLLP (in BizTalk Server 2006 R2). You must configure the receive locations to include this custom pipeline component for any orchestrations that you want to participate in ordered delivery.

Business orchestrations have to fulfill these requirements in the orchestrations:

  1. Propagate the sequence number contained in the OrderedDelivery.SequenceNumber context property to the outgoing message.
  2. Set the OrderedDelivery.PropertySchema.Destination context property to a known value such as “Gatekeeper”. This will ensure that the message is routed to the gatekeeper orchestration, which will resequence the message, because it subscribes to the Destination property.
  3. Initialize this correlation set that contains the Destination property in the send shape.
  4. The gatekeeper orchestration’s first receive shape is marked to be an activating receive shape that will initialize a correlation set that contains the OrderedDelivery.PropertySchema.Destination property.
/* Biz.odx construct message */
ProcMsg = BizMsg;

/* Propagate sequence number context property */
ProcMsg(OrderedDelivery.SequenceNumber) = BizMsg(OrderedDelivery.SequenceNumber);

/* Set destination context property to GateKeeper */
ProcMsg(OrderedDelivery.PropertySchema.Destination) = "Gatekeeper";

The MessageBox database publishes these stamped messages to orchestrations that subscribe to them. Even though the messages are published to the business orchestrations in the order received at the receive adapter, the message order may be scrambled after they are processed by business orchestrations due to the size of the messages and actual processing time. As a result, the messages are published to the MessageBox database out of order compared to the message order at the receive location.

The gatekeeper orchestration's main function is to act as a resequencer, gathering messages and ensuring outgoing order based on the stamped sequence number in each message. This is a resequencer pattern.

The gatekeeper is a singleton orchestration that is activated by the first message that matches the gatekeeper’s subscription to the OrderedDelivery.PropertySchema.Destination correlation ID. In this case the gatekeeper is initialized with the correlation value of “Gatekeeper” and all messages with this value will be routed to this singleton.

The gatekeeper orchestration performs a straightforward task of resequencing messages based on the receive order defined by the stamped sequence number. Every message received by the gatekeeper is compared with NextSequence to see if it should be sent out next. Messages that are not yet ready to be sent are added to an in-memory queue, implemented as a sorted list of key-value pairs of sequence number and message references. Note that the outgoing sequence number is maintained in memory. The durability of the sequence number is guaranteed by the fact that long-running orchestration states are durable in BizTalk Server. That is, BizTalk Server dehydrates running orchestrations when the server goes down, and restores the persisted state when the orchestrations are rehydrated.

Using a Ticket Dispenser with the Custom Pipeline

The primary function of the custom pipeline component is to stamp a sequence number as a context property in the message. The pipeline component goes out to the database table MyDB.TicketDispenser to retrieve the next available sequence that it will use to stamp the message. A database table is used because message order must be preserved across server shutdowns.

Performance is affected every time a message needs to be stamped, although this is the most straightforward way for order to be maintained during shutdowns. There are, however, ways to reduce the effect on performance. One way is to cache the sequence number in memory and periodically persist it, although this introduces timing windows where the sequence number may not be maintained across shutdowns. A more effective way is to update sequence numbers using SQL rowlock to provide the necessary atomicity for data integrity.

It is important that the ticket dispenser be placed after the pipeline disassembler in case the disassembler splits the message into one or more messages. (For example, where message 1.0 going into a disassembler comes out with messages 1.1 and 1.2.)

The acquisition of the sequence number is done in the Execute method of the pipeline component. As such, it is necessary to participate in the pipeline component’s transaction for the thread of execution. Stored procedures are used to perform the database updates called from the Execute method as follows:

CREATE PROCEDURE UpdateSequence
@ReceiveLocationName as nvarchar(50)
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    declare @val int
    declare @lastSeq int
    declare @msgInQue int
                
    UPDATE TicketDispenser set SequenceNumber =  SequenceNumber + 1
       FROM TicketDispenser WITH (ROWLOCK)
       WHERE ReceiveLocationName = @ReceiveLocationName
 
    if @@ROWCOUNT=0
    BEGIN
      SET @val=1
      SET @lastSeq=0
      SET @msgInQue=0
      INSERT into TicketDispenser(ReceiveLocationName, SequenceNumber, LastSequenceSent, MsgInQueue) values(@ReceiveLocationName, @val, @lastSeq, @msgInQue)
    END

    SELECT @val=SequenceNumber from TicketDispenser with (ROWLOCK) WHERE ReceiveLocationName = @ReceiveLocationName

    SELECT @val
END
GO

MyDB.TicketDispenser

ReceiveLocationName SequenceNumber LastSequenceSent MsgInQueue

ReceiveLocation1

888

0

0

This is the persistent table for sequence numbers stored in SequenceNumber which corresponds to a ReceiveLocationName of your choosing.

The TicketDispenser is created using the CreateTicketDispenserTable.sql script.

CREATE TABLE [TicketDispenser] (
[SequenceNumber] [int] NULL ,
[LastSequenceSent] [int] NULL ,
[MsgInQueue] [int] NULL ,
[ReceiveLocationName] [nvarchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
CONSTRAINT [PK_TicketDispenser] PRIMARY KEY  CLUSTERED 
(
[ReceiveLocationName]
)  ON [PRIMARY] 
) ON [PRIMARY]
GO

Using a Gatekeeper Orchestration

The function of the gatekeeper orchestration is to resequence messages according to the order that they arrive to ensure they are sent out in the same order. The basic algorithm is to queue out-of-order messages that are not ready to be sent out. When a message is received its sequence number is checked to see if it is equal to the last sequence number sent plus one. The message is sent out if this is the case; otherwise the message is queued. The sequence number of queued messages is maintained in memory. This is advantageous because orchestration state is preserved across orchestrations during dehydration and rehydration. Thus, message order is maintained during system restart without maintaining a separate database by leveraging orchestration state management.

The gatekeeper orchestration subscribes to the correlation initialized by the business orchestrations so that the MessageBox database publishes all post-processed messages to it. Figure 2 shows an activating receive shape (FirstBizMsg) where the OrderedDelivery.PropertySchema.Destination correlation field is initialized by the activating message. The other receive shapes will follow on this correlation to receive all messages that have been post-processed by the business orchestrations.

Figure 2. Gatekeeper First Message
Bb851740.d0bdcc79-fff9-4eb4-94c7-939dc5a4942c(en-US,BTS.10).gif

A .NET WaitQueue class implements a sorted list to store XLANG/s message references (excluding the body payload) that is not yet ready to be sent. It is important that this class perform the necessary XLANG/s message lifetime management of reference counting to keep track of whether there is any object referencing a particular message. Normally the XLANG/s runtime performs message reference counting for messages that are part of the orchestration and that are stored in the MessageBox database. In this case, the message is being referenced in .NET code outside of the runtime.

The .NET WaitQueue class method RemoveMessage() is called when a message is found in the queue that is the next sequenced message to be sent out. The RemoveMessage() calls the dispose method of the message to signal to the runtime that this instance is done with the message.

Figure 3 shows the main loop of the gatekeeper orchestration that receives and processes all messages after the first activating message. There are two branches to this loop. One branch checks for message order before the message is sent out and the other branch handles the control message to shut down this singleton.

Figure 3. Gatekeeper Main Loop
Bb851740.beb28f88-a441-45fb-992f-f7aabec5fe95(en-US,BTS.10).gif

Message order is determined by inspecting the stamped sequence number of each message against the sequence number of the last message sent. There is additional logic to check the WaitQueue every time a message is sent. In other words, every time a message is sent out it checks the WaitQueue to see if the next sequence number is queued. Since this is a sorted list it allows potentially for a batch of sequential messages to all be sent one after another. For example, if the next message to be sent in the sequence is 55, while messages 56 through 100 are queued up, when message 55 eventually arrives and is sent out, the logic path will ensure that messages 56 through 100 are sent out in quick succession to flush the queue.

This gatekeeper orchestration is able to respond to control messages to shut down. Upon receiving the message to shut down it checks to see if there are messages in the WaitQueue. If there are none it writes out the last sequence sent before it shuts down. This value is used to initialize the next sequence number when the gatekeeper is reactivated so that order can be maintained across shutdowns of the singleton gatekeeper. If there are messages in the WaitQueue the gatekeeper suspends and waits for user intervention.

Error Handling

We will consider a couple of error conditions and possible ways to handle them.

  • Hole in the sequence message: This is a scenario where the gatekeeper is waiting for a message with a specific sequence number and the message never arrives. This can happen if a business process consumes one of the messages and does not forward it. There is no easy algorithmic solution to this problem. Depending on the application, a practical approach is to skip the message after a designated time-out value, and to log the event. To accommodate for known latency a retry count can be used in conjunction with the time-out value before a message is skipped.
  • Messages held in the gatekeeper orchestration when it is shutting down: Since XLANG/s messages have to exist within the context of an orchestration, routing messages to a database does not work because the message lifetime context is lost and the runtime does not provide a mechanism to load stored messages. It is possible to route messages to a surrogate orchestration while the gatekeeper restores itself (where it was perhaps shut down due to memory pressure), and to receive the messages back when it comes back up. The restored gatekeeper also reloads the WaitQueue together with the rest of the variable states like next sequence number. This solution, however, is fairly complicated to use as a general solution to handle all such situations.

Design Tradeoffs

We will consider some tradeoffs for the design and implementation described in this white paper.

  • Why is the gatekeeper implemented as an orchestration instead of a send custom pipeline component?
    One clear reason against a pipeline implementation is a deadlock that can occur when the pipeline instance gets throttled while BizTalk Server is waiting for a message with the next allowed sequence. Once throttled even when the specific message finally arrives, BizTalk Server does not give the thread of execution to the pipeline because it has been throttled out already. This happens when BizTalk Server is under high memory pressure and the message queue backlog is high.
    By contrast, when you use an orchestration you can leverage the built-in memory management mechanisms in dehydration and rehydration to manage state. You also do not have to maintain a separate persistent store required by a pipeline component for system recovery. Furthermore, orchestrations provide flexibility to model business semantics in the orchestration business logic. An example is correlating messages and patient IDs.
  • Database configuration
    The custom pipeline component uses a database that is configured in the BizTalk Server btsntsvc.exe.config file. Optionally, the SSO database can be used to allow central configuration when deploying to a server farm. This enhances security with an encrypted connect string and password.
  • Content-based ordering (CBO)
    The ticket dispenser custom pipeline component may be implemented with an orchestration doing in-memory sequence stamping. There is no need to maintain a user database for sequence number persistence, as this is taken care of by orchestration state management in the runtime.
  • Gatekeeper
    The advantages of implementing the resequencer (gatekeeper) as an orchestration are that it increases performance due to no database cost, and because of the ability to leverage the orchestration correlation feature for routing.
  • Ticket Dispenser
    The disadvantages of implementing the ticket dispenser as an orchestration as opposed to a custom pipeline component, as described in this paper, is that it increases known singleton state bloat and management issues.

Advanced Modeling

The following considerations explain more advanced implementations of a gatekeeper orchestration to further improve ordered delivery for specific business requirements.

  • Multiple Receive Locations
    The sample in this white paper demonstrates ordered delivery between a pair of receive and send ports. However, it is common to have more than one pair of receive and send ports. As previously discussed, it is possible to support multiple pairs of ports in the TicketDispenser table, where each row represents a pair of ports. To fully support this feature there needs to be a gatekeeper singleton for each pair of ports. There is an elegant way to achieve this by making the correlation value of each singleton the receive location name.
  • Relative Ordering
    It can be argued that businesses need relative ordering of all messages pertaining to a single entity such as a patient ID more than absolute ordering of all messages going through the system for all patients.
    This can be easily done by activating a singleton gatekeeper for each patient and correlating messages based on patient ID. The added benefit of the WaitQueue is smaller since it is only dealing with messages for one patient. The lifetime of the orchestration is defined by the patient check-in that activates the singleton and patient check-out which ends the orchestration. This pattern is suitable for other applications like stock trades, processing a buy or sell order through order confirmation, selling or buying orders, funds settlement, and closing the order.
  • Sequence Wraparound
    The size of the sequence number in this implementation is an integer. The sequence number will at some point wrap around back to one. This implementation assumes the custom pipeline and ticket dispenser orchestration will be recycled before reaching the integer size boundary before wraparound occurs.
    A more robust design is to add a batch field to the context property schema which contains a GUID. The ticket dispenser acquires a batch GUID and stamps messages with sequence numbers within the integer range. When the maximum integer is reached a new GUID is acquired to denote a new batch and the sequence number starts at one. Consequently, the gatekeeper orchestration has to compute message sequence numbers within the context of its batch.
    In the custom pipeline, when the maximum size in the integer range is reached, a new batch GUID is generated to be used in the batch field and the sequence number begins with one. It is not necessary to synchronize sequence numbers between the ticket dispenser and gatekeeper in this scheme. The custom pipeline stamps batch and sequence numbers and the gatekeeper orchestration processes the next sequence based on the sequence number in each batch.

Performance

This section contains information about performance and system configuration for the solution. For background information about BizTalk Server performance, see http://go.microsoft.com/fwlink/?LinkId=93137.

Maximum sustainable throughput

  • 4.3 msg/sec in a single-computer configuration
  • 11 msg/sec in a five-computer configuration

Hardware configurations

  • Dual Proc Xeon 3.40 GHz, 2 GB of RAM
  • HD (SCSI)

Message size

  • Message size varied between 0.5 KB and 18 KB with an average message size of 1 KB.

Software configuration

  • Microsoft Windows Server 2003, SQL Server™ 2005
  • BizTalk Server 2006 R2

MSMQ adapter configuration

  • Receive
    • Batch size 1
    • Transactional
    • Ordered processing
  • Send
    • Transactional
    • Retry count 0
    • Ordered Delivery
    • Stop sending on message failure
    • Enabled routing for failed messages

BizTalk Server configuration

  • Two hosts - one to host the gatekeeper and other for the MSMQ send handler
  • Single pair of transactional private MSMQ queues to send and receive test data

Multicomputer Configuration (five computers)

  • SQL Server 1: BizTalk Server databases
  • SQL Server 2: MyDB
  • Receive computer: local MSMQ receive queue, MSMQ receive handler, BizTalkServerApplication host instance, MyBiz orchestrations
  • GateKeeper computer: GKHost BizTalk host instance, MSMQ send handler, Gatekeeper orchestration
  • Monitoring computer: Send test data and running Perfmon counters/logs

The following sample has a test driver shown in Figure 4 and three business orchestrations, one of which is shown in Figure 5. The test driver when started will construct three XML messages, each with different message_type values of ADT1, ADT2, and ADT3 (HL7 message types). One of the messages is as shown. The test driver sends five messages of each type to the respective locations where three business orchestrations will pick them up.

BusinessMessage.xsd

 <?xml version="1.0" encoding="utf-16" ?> 
 <xs:schema xmlns:ns0="http://OrderedDelivery.PropertySchema.PropertySchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://OrderedDelivery.BusinessMessage" targetNamespace="http://OrderedDelivery.BusinessMessage" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:annotation>
   <xs:appinfo>
         <b:imports xmlns:b="http://schemas.microsoft.com/BizTalk/2003">
           <b:namespace prefix="ns0" uri="http://OrderedDelivery.PropertySchema.PropertySchema"     location=".\propertyschema.xsd" /> 
      </b:imports>
      </xs:appinfo>
          </xs:annotation>
       <xs:element name="BusinessMessage">
     <xs:annotation>
   <xs:appinfo>
  <b:properties>
   <b:property distinguished="true" xpath="/*[local-name()='BusinessMessage' and namespace-uri()='http://OrderedDelivery.BusinessMessage']/*[local-name()='MessageId' and namespace-uri()='']" /> 
  <b:property name="ns0:GateKeeperStop" xpath="/*[local-name()='BusinessMessage' and namespace-uri()='http://OrderedDelivery.BusinessMessage']/*[local-name()='GateKeeperStop' and namespace-uri()='']" /> 
  <b:property name="ns0:TestDriverStart" xpath="/*[local-name()='BusinessMessage' and namespace-uri()='http://OrderedDelivery.BusinessMessage']/*[local-name()='TestDriverStart' and namespace-uri()='']" /> 
  <b:property name="ns0:MessageType" xpath="/*[local-name()='BusinessMessage' and namespace-uri()='http://OrderedDelivery.BusinessMessage']/*[local-name()='MessageType' and namespace-uri()='']" /> 
  </b:properties>
  </xs:appinfo>
  </xs:annotation>
  <xs:complexType>
  <xs:sequence>
  <xs:element name="PatientId" type="xs:int" /> 
  <xs:element name="Symptoms" type="xs:string" /> 
  <xs:element name="MessageId" type="xs:int" /> 
  <xs:element name="GateKeeperStop" type="xs:boolean" /> 
  <xs:element name="TestDriverStart" type="xs:boolean" /> 
  <xs:element name="MessageType" type="xs:string" /> 
  </xs:sequence>
  </xs:complexType>
  </xs:element>
 </xs:schema>

BusinessMessage_1.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>false</GateKeeperStop> 
  <TestDriverStart>false</TestDriverStart> 
  <MessageType>ADT1</MessageType> 
</ns0:BusinessMessage>

BusinessMessage_1.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>false</GateKeeperStop> 
  <TestDriverStart>false</TestDriverStart> 
  <MessageType>ADT1</MessageType> 
</ns0:BusinessMessage>

BusinessMessage_2.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>false</GateKeeperStop> 
  <TestDriverStart>false</TestDriverStart> 
  <MessageType>ADT2</MessageType> 
</ns0:BusinessMessage>

BusinessMessage_3.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>false</GateKeeperStop> 
  <TestDriverStart>false</TestDriverStart> 
  <MessageType>ADT3</MessageType> 
</ns0:BusinessMessage>

StartTestDriver.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>false</GateKeeperStop> 
  <TestDriverStart>true</TestDriverStart> 
  <MessageType>Control</MessageType> 
</ns0:BusinessMessage>

StopGatekeeper.xml

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId> 
  <Symptoms>Symptoms_0</Symptoms> 
  <MessageId>10</MessageId> 
  <GateKeeperStop>true</GateKeeperStop> 
  <TestDriverStart>false</TestDriverStart> 
  <MessageType>Control</MessageType> 
</ns0:BusinessMessage>

PropertySchema.xsd

<?xml version="1.0" encoding="utf-16" ?> 
  <xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://OrderedDelivery.PropertySchema.PropertySchema" targetNamespace="http://OrderedDelivery.PropertySchema.PropertySchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:annotation>
  <xs:appinfo>
  <b:schemaInfo schema_type="property" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" /> 
  </xs:appinfo>
     </xs:annotation>
         <xs:element name="Destination" type="xs:string">
   <xs:annotation>
  <xs:appinfo>
  <b:fieldInfo propertyGuid="4be0511f-ed6d-4f15-a5cd-5f52f6c3f2f9" propSchFieldBase="MessageContextPropertyBase" /> 
  </xs:appinfo>
  </xs:annotation>
   </xs:element>
 <xs:element name="GateKeeperStop" type="xs:boolean">
<xs:annotation>
<xs:appinfo>
  <b:fieldInfo propertyGuid="46e85930-7176-4439-a6ec-e9794e6b923c" propSchFieldBase="MessageContextPropertyBase" /> 
  </xs:appinfo>
  </xs:annotation>
  </xs:element>
   <xs:element name="MessageId" type="xs:int">
  <xs:annotation>
  <xs:appinfo>
  <b:fieldInfo propertyGuid="72440ecc-6734-44a8-ba8d-a5c45bff3e7b" /> 
  </xs:appinfo>
  </xs:annotation>
  </xs:element>
   <xs:element name="TestDriverStart" type="xs:boolean">
  <xs:annotation>
  <xs:appinfo>
  <b:fieldInfo propertyGuid="6b0a87e0-0db2-4c36-aae8-bcb2415e37f8" /> 
  </xs:appinfo>
  </xs:annotation>
  </xs:element>
   <xs:element name="MessageType" type="xs:string">
  <xs:annotation>
  <xs:appinfo>
  <b:fieldInfo propertyGuid="8b13396a-6a6c-4bd3-a860-97b52fa05da9" propSchFieldBase="MessageContextPropertyBase" /> 
  </xs:appinfo>
  </xs:annotation>
  </xs:element>
  </xs:schema>
Figure 4. Test Driver
Bb851740.2a5b43e3-8c9d-4f16-9bcb-24c6f5d8d7b7(en-US,BTS.10).gif

The receive locations for each of the business orchestrations are configured with the custom pipeline component in order to stamp a sequence number.

Bb851740.note(en-US,BTS.10).gifNote
PatientId and Symptoms are business elements to simulate a healthcare application, and are not used in the solution to determine order.

<ns0:BusinessMessage xmlns:ns0="http://OrderedDelivery.BusinessMessage">
  <PatientId>10</PatientId>
  <Symptoms>Symptoms_0</Symptoms>
  <MessageId>10</MessageId>
  <GateKeeperStop>false</GateKeeperStop>
  <TestDriverStart>false</TestDriverStart>
  <MessageType>ADT1</MessageType>
</ns0:BusinessMessage>

The business orchestrations propagate the context properties and initialize the message correlation to “Gatekeeper” so that the message will be published to the gatekeeper for resequencing.

Figure 5. Business Orchestration
Bb851740.cedf42e9-71de-4a1a-bc17-82d8f24316ab(en-US,BTS.10).gif

This section gives procedures for building and deploying the sample solution.

Setting Up the Solution

Use the following procedure to set up the solution.

  1. Unzip the solution to your local computer, and place the sample solution in the root directory of C:\.

  2. Create the following directories:

    • C:\Input
    • C:\Output

Building and Deploying the Solution

Use the following solution to build and deploy the solution.

  1. Create a strong name assembly file called Sequence.snk and place it in the sample root directory C:\OrderedDelivery.

    You can do this at a Microsoft Visual Studio® command prompt: sn.exe –k Sequence.snk.

  2. Using Visual Studio 2005, open OrderedDelivery.sln.

  3. Build the Helper project:

    1. Right-click the Helper project to add C:\OrderedDelivery\Sequence.snk to the project.
    2. Specify in the project properties that Sequence.snk is used as the strong name key file to sign the assembly.
    3. Build Helper project which will add Microsoft.Samples.BizTalk.OrderedDelivery.Helper.dll to the global assembly cache (GAC).
  4. Build the TicketDispenser project:

    1. Right-click the TicketDispenser project to add C:\OrderedDelivery\Sequence.snk to the project.
    2. Specify in the project properties that Sequence.snk is used as the strong name key file to sign the assembly.
    3. Build TicketDispenser project which will add Microsoft.Samples.BizTalk.OrderedDelivery.Pipeline.CustomComponent.TicketDispenser.dll to the global assembly cache (GAC).
  5. Build the Schemas project:

    1. Add a reference and browse to C:\Program Files\Microsoft BizTalk Server 2006 to add Microsoft.BizTalk.DefaultPipelines.dll to the project.
    2. Build the Schemas project (right-click the project and then select Build).
  6. Build the Pipeline project:

    1. Open TicketDispenserReceivePipeline.btp.
    2. In the Toolbox pane, right-click BizTalk Pipeline Components, and then select Choose Items.
    3. Click the BizTalk Pipeline Components tab and browse to C:\OrderedDelivery\TicketDispenser\bin\release (or debug), and then select Microsoft.Samples.BizTalk.OrderedDelivery.Pipeline.CustomComponent.TicketDispenser.dll. Select the check box so that it will be displayed.
    4. Drag the Ticket Dispenser Component to the Validate stage in the Pipeline designer.
    5. Build the Pipeline project (right-click the project and then select Build).
  7. Build the Business project (right-click the project and then select Build).

  8. Build the GateKeeper project (right-click the project and then select Build).

  9. Build the TestDriver project (right-click the project and then select Build).

  10. Ensure that the BizTalk projects deployment configuration database property matches the actual BizTalk group configuration database (shown in the BizTalk Server Administration Console).

  11. Deploy the Schemas project (right-click the project, and then select Deploy). View the following schemas in the BizTalk Server Administration Console Schema node:

    • OrderedDelivery.TicketDispenserPropertySchema
    • OrderedDelivery.PropertySchema
    • OrderedDelivery.BusinessMessage
  12. Deploy the Pipeline project (right-click the project, and then select Deploy). View the pipeline component in the BizTalk Administration Console Pipelines node:

    • Pipeline.TicketDispenserReceivePipeline
  13. Deploy the Business project (right-click the project, and then select Deploy). View the orchestrations in the BizTalk Server Administration Console Orchestrations node:

    • OrderedDelivery.MyBiz
    • OrderedDelivery.MyBiz2
    • OrderedDelivery.MyBiz3
  14. Deploy the Gatekeeper project (right-click the project, and then select Deploy). View the orchestration in the BizTalk Server Administration console Orchestrations node:

    • OrderedDelivery.Gatekeeper
  15. Deploy TestDriver project (right-click the project, and then select Deploy). View the orchestration in the BizTalk Server Administration console Orchestrations node:

    • OrderedDelivery.TestDriver

Configuring the Database

Use the following procedure to configure the database.

  1. Create MyDB database in SQL Server.

  2. Create TicketDispenser table in MyDB using CreateTicketDispenser.sql.

  3. Import stored procedures defined in Sequence.sql into MyDB.

Configuring BizTalk Server

Use the following procedure to configure BizTalk Server.

  1. Add the following to the BizTalk configuration file btsntsvc.exe.config:

    <appSettings>
    <add key="OD_ConnectString" value="Data Source=localhost;Integrated Security=SSPI;Initial Catalog=MyDB;Enlist=false;" />
    <add key="OD_ReceiveLocation" value="ReceiveLocation1"/>
    <add key="OD_Biz1_Message" value="C:\OrderedDelivery\Data\BusinessMessage_1.xml"/>
    <add key="OD_Biz2_Message" value="C:\OrderedDelivery\Data\BusinessMessage_2.xml"/>
    <add key="OD_Biz3_Message" value="C:\OrderedDelivery\Data\BusinessMessage_3.xml"/>
    </appSettings>
    
  2. Restart the BTSNTSvc service.

  3. Open the BizTalk Server Administration console and navigate to BizTalk Application 1 where the solution was deployed.

  4. Right-click BizTalk Application 1 and select Configure to create the following ports and locations:

    Create TestDriver Receive port

    1. Right-click Receive ports to create a One-Way Receive port naming it TestDriverReceivePort.
    2. Click Receive Locations, and then click New to specify:
      Name: TestDriverReceiveLocation
      Transport Type: File
      Receive Folder: C:\input
      File mask: *.xml
      Receive pipeline: XMLReceive

    Create the MSMQT adapter

    1. Right-click Adapter in Platform Settings.
    2. Name the adapter MSMQT.
    3. Choose MSMQT from the adapter drop-down list.
    4. Stop the MSMQ service through the Device Manager MMC to ensure that messages are consumed only by MSMQT.
    5. Restart the BizTalk host instance for MSMQT configuration to take effect.

    Create MSMQT receive ports and locations

    1. Right-click Receive ports to create a One-Way Receive port naming it BusinessReceiveMSMQTPort.
    2. Right-click Receive Locations, and then click New to specify the following:
      Name: BusinessReceiveMSMQTPortLocation
      Transport Type: MSMQT
      Configure Queue name: inQueue
      Receive pipeline: TicketDispenserReceivePipeline

    Create the MSMQT send port for the TestDriver

    • Right-click Send port to create a new Static One-way Send Port, and then specify the following:
      Port Name: TestDriverMSMQTSendPort
      Transport: MSMQT
      Configure Destination Queue: DIRECT=OS:localhost\Private$\inQueue
      Send pipeline: XMLTransmit

    Create a send port for the gatekeeper

    • Right-click Send port to create a new Static One-way Send Port and specify the following:
      Port Name: GatekeeperSendPort
      Transport: File
      URI: C:\output
      File name: %messageID%\*.xml
      Send pipeline: XMLTransmit
    • Click Transport Advanced Options and select Ordered Delivery and Stop sending subsequent messages on current message failure.

    Bind orchestrations MyBiz, MyBiz1, MyBiz2

    • Host to BizTalkServerApplication
    • ReceiveMsg to BusinesReceiveMSMQTPort

    Bind Gatekeeper orchestration

    • Host to BizTalkServerApplication
    • GatekeeperSendMsg to GatekeeperSendPort

    Bind TestDriver orchestration

    • Host to BizTalkServerApplication
    • Port_StartTest to TestDriverReceivePort
    • Port_MSMQT to TestDriverMSMQTSendPort

Running the Solution

Use the following procedure to run the solution.

  1. Start the BizTalk application using the BizTalk Server Administration console.

  2. Attach another instance of Visual Studio to the btsntsvc.exe process to see trace output.

  3. Drop StartTestDriver.xml into C:\Input.

  4. You should see a max of 15 instances of biz.odx and one gatekeeper orchestration.

  5. You should see 15 files in C:\output directory when run is completed.

  6. Verify order in the trace output.

  7. Verify the TicketDispenser table shows sequence number of 15 in the SQL Query Analyzer by typing select * from TicketDispenser.

Stopping the Gatekeeper Orchestration

Use the following procedure to stop the gatekeeper orchestration.

  1. Drop StopGateKeeper.xml into C:\Input.

  2. Verify that the BizTalk Server hub view shows there is no gatekeeper orchestration instance.

Verifying Order Durability

Use the following procedure to verify order durability.

  • To see that order is maintained across server shutdown and from one instance of the gatekeeper to another, rerun the test repeating “Running the solution”. You can also power cycle the computer during a run.

This is a test run trace output from the gatekeeper orchestration demonstrating how it resequences out-of-order messages.

// The seventh message received by the Receive adapter is the first message that activates
// Gatekeeper.  Since the first message to go out is sequence 1 this message is queued.
Init-MessageType: ADT1
Init-ReceivedSequenceNumber: 7
Init-NextSequence: 1
StartAddMsg-SequenceNumber: 7
StartAddMsg-NextSequence: 1
StartAddMsg-QCount: 1

// Sequence 2 is queued
BizMsg-MessageType: ADT1
BizMsg-ReceivedSequenceNumber: 2
AddMsg-SequenceNumber: 2
AddMsg-NextSequence: 1
AddMsg-QCount: 2

// Sequence 4 is queued
BizMsg-MessageType: ADT1
BizMsg-ReceivedSequenceNumber: 4
AddMsg-SequenceNumber: 4
AddMsg-NextSequence: 1
AddMsg-QCount: 3

// Ah! Sequence 1 came in, let’s send it out and check for 2, found it, send it out
BizMsg-MessageType: ADT2
BizMsg-ReceivedSequenceNumber: 1
SendMessage-SentSequenceNumber: 1
SetNextMsg-NextSequence: 2
IncNextSeq-SentQueuedSequenceNumber: 2
IncNextSeq-QCount: 2
FlushLoopDone

// Sequence 5 is queued
BizMsg-MessageType: ADT2
BizMsg-ReceivedSequenceNumber: 5
AddMsg-SequenceNumber: 5
AddMsg-NextSequence: 3
AddMsg-QCount: 3

// Sequence 9 is queued
BizMsg-MessageType: ADT2
BizMsg-ReceivedSequenceNumber: 9
AddMsg-SequenceNumber: 9
AddMsg-NextSequence: 3
AddMsg-QCount: 4

// Sequence 3 is here, send it out, flush 4 and 5
BizMsg-MessageType: ADT3
BizMsg-ReceivedSequenceNumber: 3
SendMessage-SentSequenceNumber: 3
SetNextMsg-NextSequence: 4
IncNextSeq-SentQueuedSequenceNumber: 4
IncNextSeq-QCount: 3
IncNextSeq-SentQueuedSequenceNumber: 5
IncNextSeq-QCount: 2
FlushLoopDone

// Sequence 8 is queued
BizMsg-MessageType: ADT1
BizMsg-ReceivedSequenceNumber: 8
AddMsg-SequenceNumber: 8
AddMsg-NextSequence: 6
AddMsg-QCount: 3

// Send sequence 6, 7, 8, 9 out
BizMsg-MessageType: ADT3
BizMsg-ReceivedSequenceNumber: 6
SendMessage-SentSequenceNumber: 6
SetNextMsg-NextSequence: 7
IncNextSeq-SentQueuedSequenceNumber: 7
IncNextSeq-QCount: 2
IncNextSeq-SentQueuedSequenceNumber: 8
IncNextSeq-QCount: 1
IncNextSeq-SentQueuedSequenceNumber: 9
IncNextSeq-QCount: 0
FlushLoopDone

// Sequence 10 came and it is the next one to go out
BizMsg-MessageType: ADT3
BizMsg-ReceivedSequenceNumber: 10
SendMessage-SentSequenceNumber: 10
SetNextMsg-NextSequence: 11
FlushLoopDone

// Sequence 11 came and it is the next one to go out
BizMsg-MessageType: ADT2
BizMsg-ReceivedSequenceNumber: 11
SendMessage-SentSequenceNumber: 11
SetNextMsg-NextSequence: 12
FlushLoopDone

// Sequence 12 came and it is the next one to go out
BizMsg-MessageType: ADT3
BizMsg-ReceivedSequenceNumber: 12
SendMessage-SentSequenceNumber: 12
SetNextMsg-NextSequence: 13
FlushLoopDone

// Sequence 13 came and it is the next one to go out
BizMsg-MessageType: ADT1
BizMsg-ReceivedSequenceNumber: 13
SendMessage-SentSequenceNumber: 13
SetNextMsg-NextSequence: 14
FlushLoopDone

// Sequence 14 came and it is the next one to go out
BizMsg-MessageType: ADT2
BizMsg-ReceivedSequenceNumber: 14
SendMessage-SentSequenceNumber: 14
SetNextMsg-NextSequence: 15
FlushLoopDone

// Sequence 15 came and it is the next one to go out
BizMsg-MessageType: ADT3
BizMsg-ReceivedSequenceNumber: 15
SendMessage-SentSequenceNumber: 15
SetNextMsg-NextSequence: 16
FlushLoopDone

  1. C:\OrderedDelivery\Business contains simulated business orchestrations each processing its own message type.
    • Biz.odx – Filter expression subscribing to MessageType=ADT1.
    • Biz2.odx – Filter expression subscribing to MessageType=ADT2.
    • Biz3.odx – Filter expression subscribing to MessageType=ADT3.
  2. C:\OrderedDelivery\Gatekeeper\Gatekeeper.odx is the gatekeeper orchestration.
  3. C:\OrderedDelivery\Pipeline\ TicketDispenserReceivePipeline.btp is the pipeline component.
  4. C:\OrderedDelivery\TicketDispenser\Sequencing.cs is the custom pipeline component code.
  5. C:\OrderedDelivery\Helper contains helper classes.
    • WaitQueue.cs – In-memory queue of messages.
    • Persistence.cs – Database read/write methods.
    • Config.cs – Configuration class.
    • Tracing.cs – Trace diagnostic class.
  6. C:\OrderedDelivery\Schemas contains common schema files for the orchestrations
    • BusinessMessage.xsd
    • PropertySchema.xsd
    • TicketDispenserPropertySchema.xsd
  7. C:\OrderedDelivery\TestDriver\TestDriver.odx is the test driver orchestration.
  8. C:\OrderedDelivery\SQL
    • CreateTicketDispenserTable.sql – Creates ticket dispenser table.
    • Sequence.sql – Stored procedures used by ticket dispenser.

Copyright

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred

© 2007 Microsoft Corporation. All rights reserved.

Microsoft, BizTalk, Visual Studio, and Windows Server are trademarks of the Microsoft group of companies.

All other trademarks are property of their respective owners.

Show:
© 2014 Microsoft