Writing Effective BizTalk Server Adapters

Microsoft Corporation

Published: July 2005

Applies to: BizTalk Server 2004

Summary: The Writing Effective BizTalk Server Adapters document contains information to help you understand the issues you may encounter when writing Microsoft® BizTalk Server® adapters, and provides guidelines for resolving these issues. (32 printed pages)

Many people, both within Microsoft and in third-party companies, have written successful adapters for Microsoft® BizTalk Server®. This task can be difficult, and this paper is intended to present some of the "tricks of the trade" that these developers have learned, in the hope that these will help others avoid common problems.

This document is structured as a set of issues that programmers face when writing adapters, and provides guidelines for resolving these issues. These guidelines can be read in any order, although you may find that reading the whole document first will make the individual guidelines easier to understand.

Much of the complexity in writing adapters for BizTalk Server is centered on the problems of batched operations and that is what we will look at first.

Adapters commonly support sending messages to and receiving messages from BizTalk Server:

  • Send side operations occur when information is being sent by BizTalk Server to the protocols supported by the adapter.
  • Receive Side operations occur when the adapter receives information from the outside world and passes it into BizTalk Server.

Adapters in BizTalk Server perform a number of fundamental operations when receiving and sending messages. These operations are expressed as methods on an IBTTransportBatch batch object. With one exception, the methods take a message as a parameter. These operations described in the following sections.

Item 1: Receive Side Operations

  • One-Way Submit: void SubmitMessage(IBaseMessage msg). The adapter can submit a message to BizTalk Server.
  • Suspend: void MoveToSuspendQ(IBaseMessage msg). The adapter can suspend or move a message to the suspend queue when certain failures occur after submission.
  • Submit Request: void SubmitRequestMessage(IBaseMessage requestMsg, string correlationToken, [MarshalAs(UnmanagedType.Bool)]bool firstResponseOnly, DateTime expirationTime, IBTTransmitter responseCallback). This operation allows a receive adapter to submit a message to BizTalk Server in a request-response pair.

Item 2: Send Side Operations

  • Resubmit: void Resubmit(IBaseMessage msg, DateTime timeStamp). The adapter can resubmit a message after transmission failures occur. The resubmit operation requests that BizTalk Server schedules the message for transmission at a later time.
  • Move to Next Transport : void MoveToNextTransport(IBaseMessage msg). The adapter can move the message to the next transport after all retries have been exhausted. This operation requests that a message is transmitted using the backup transport for that message.
  • Suspend: void MoveToSuspendQ(IBaseMessage msg). The adapter can move the message to the suspend queue when no backup transport is present
  • Delete: void DeleteMessage(IBaseMessage msg) The adapter can delete the message after a successful transmission. Deleting a message in BizTalk Server is the way that a send adapter tells BizTalk Server that the adapter is done with the message. Generally the SubmitResponse operation is done in the same batch as the Delete operation.
  • Submit Response: void SubmitResponseMessage(IBaseMessage solicitMsgSent, IBaseMessage responseMsgToSubmit). The adapter can submit an immediate response back to BizTalk Server. This operation includes the original message in the call along with the response so that BizTalk Server can correlate them. The response message does not need to be the true external system response message, which might arrive later.

Item 3: Messages

A BizTalk Server message is a packet of data that arrives in BizTalk Server or is sent from BizTalk Server. Everything going into and coming out of BizTalk Server is a message. Examples include HTTP posts, FTP files, e-mail, MQSeries messages, and SQL result sets. These are all made into messages in the BizTalk Server world.

A message is made up of a header section and a body section:

  • Headers are simple name-value pairs. The names are all drawn from various property schemas. The system provides some core schemas and BizTalk Server applications can add their own. Headers are copied into memory when a message is processed.
  • The body of a message has one or more parts. Each part is a stream of bytes. The body of the message is never copied into memory at one time.

The basic data model for a message is inspired by the Internet e-mail protocol MIME.

Item 4: Batches

When your adapter has a series of operations to perform at one time, you should batch these operations to optimize performance.

Programmatically, message batches are collections of operations with associated messages.

Item 5: Transactions

Batches are associated with transactions. For every batch, there is a single transaction. The adapter may create the transaction and associate it with the batch (in which case we call it a transactional batch) or it can leave it to BizTalk Server to manage internally by providing a NULL transaction object with the batch (in which case we call it a normal batch). BizTalk Server itself creates an internal batch object in this case, but this batch object will not be accessible to the adapter.

The idea of a "transactional Batch" can be somewhat confusing, because all batches, inside of BizTalk Server, are considered transactions.

  • Transactional Batches ― If the adapter wants to involve the external system in the transaction in any way (for example, SQL Server in the case of the SQL Adaptor) then it must specifically create a transaction object. A transactional batch is a batch for which the adapter provides the transaction object that BizTalk Server will use.
    For the purposes of this paper, a transaction is a true Microsoft Distributed Transaction Coordinator (MSDTC) COM+ transaction.
  • Transactional Adapter ― A transactional adapter is an adaptor that makes use of transactional batches. A "Non-transactional adapter" uses normal batches.

BizTalk Server uses batching to:

  • Amortize the cost of the transaction across many messages.
  • Increase speed by reducing the internal number of database round trips.
  • Make more efficient use of the BizTalk Server thread pool by using the BizTalk Server Asynchronous API.

A batch is a unit of work that is atomic; that is, it either succeeds or fails. If one operation in a batch succeeds but another operation fails, all the operations that make up a batch are invalidated and must be repeated.

This means that an adapter must do two things in response to a failed batch:

  • Determine what to do with the failed message
  • Re-submit the successful messages

Item 1: Types of Batches

There are two kinds of batches defined previously in this paper: transactional batches and normal batches. These types are used in the following ways:

  • Normal batches ― Not all adapters need to manage transactions externally. The FILE adapter is an example of an adapter that does not require access to the transaction, because the external file operations it manages are not transactional. In this case, the adapter does not provide a transaction object to BizTalk Server. If the adapter does not provide a transaction object to BizTalk Server as part of a batch of messages, BizTalk Server creates an internal transaction for it automatically. This can be thought of as a "normal" batch.
  • Transactional batches ― Other adapters (such as the SQL adapter supplied with BizTalk Server) must coordinate an external SQL Server transaction with an internal BizTalk Server transaction. To do this, the adapter needs access to the BizTalk Server transaction object. A transaction object is created and and associated with the batch before the batch is submitted to BizTalk Server. This is a transactional batch. By supplying your own transaction object, you can achieve the "guaranteed, once and once only", delivery of data into and out of BizTalk Server.
    Transactional database adapters like the SQL adapter that shipped with BizTalk Server 2004 have the potential for deadlocks in the external database because of the single transaction used for the batch. This is why the SQL adapter hard-coded its batch size to one.
An adapter provides BizTalk Server with a transaction to ensure that either BizTalk Server or the external system has a record of the data. Because only one or the other has this record, once and once only delivery is assured.

Item 2: Processing Errors

Error processing should be associated with an individual operation and not with the batch that contains the operation. Performance issues aside, the batching of messages should be invisible to the user of the adapter. This means that the failure of one operation in a batch should not affect any other operation in any way. However, batches are atomic, so the failure of one message results in an error for the batch, and no operations are processed.

Your write the code that is responsible for handling the error, re-submitting the successful messages, and suspending the unsuccessful ones. Fortunately, BizTalk Server provides a detailed error structure that enables your adapter to determine the specific operation that failed, and to construct further batches in which the successful operations are resubmitted and the unsuccessful ones suspended.

The final durable state of the operation should not be affected by the batching that happened to go on within the adapter.

Item 3: Reporting Failure

If you are suspending a message, you must provide failure information to BizTalk Server from the previous message context. BizTalk Server provides error reporting capabilities using the SetErrorInfo method on both the IBaseMessage interface as well as on ITransportProxy:

  • When a failure occurs while processing a message, set the exception using SetErrorInfo(exception) on the message to be suspended. This way, the engine preserves this error with the message for later diagnosis, and logs it to the event log to alert the administrator.
  • If you encounter an error during initialization or internal bookkeeping (not during message processing) you should call SetErrorInfo(exception) on the ITransportProxy pointer passed to you during initialization. If your adapter is based on a BaseAdapter implementation, you should always have access to this pointer. Otherwise, you should be certain that you cache it.

Reporting an error with either of these methods results in the error getting to the event log. It is important that you associate the error with the related message if you are able to do so.

Item 4: Confirming Success

When you write an adapter that creates a transaction object and hands it to BizTalk Server, you are accepting responsibility for writing code that does several things:

  • Decides the final outcome of the batch operation: to either commit or end the transaction.
  • Informs BizTalk Server of the final outcome by calling the method, DTCConfirmCommit()

The adapter must inform BizTalk Server about the final outcome of the transaction in order to maintain its internal tracking data. The adapter informs BizTalk Server of the outcome by calling DTCConfirmCommit. If the adapter does not do this, a significant memory leak occurs.

Item 5: Avoiding Race Conditions

The two tasks listed above (resolve errors and decide the final outcome) seem simple enough, but in fact they rely on information from different threads:

  • The adapter processes errors based on information passed by BizTalk Server to the BatchComplete callback in the adapter. This callback is on the adapter's thread.
  • DTCConfirmCommit is a method on the IBTDTCCommitConfirm object. An instance of the IBTDTCCommitConfirm object is returned by the batch IBTTransportBatch::Done() call. This instance is on the same thread as the IBTTransportBatch::Done() call, which is different from the adapter's thread.
  • For every call that adapter makes to IBTTransportBatch::Done() there is a corresponding callback BatchComplete() that is called by the messaging engine in a separate thread to report the result of the batch submission. In BatchComplete() the adapter needs to commit or roll back the transaction based on whether the batch passed or failed. In either case, the adapter should then call DTCConfirmCommit() to report the status of the transaction.

A possible race exists because the adapter’s implementation of BatchComplete can assume that the IBTDTCCommitConfirm object returned by IBTTransportBatch::Done() is always available when BatchComplete executes. However, BatchComplete() can be called in a separate messaging engine thread, even before IBTTransportBatch::Done() returns. It is possible that when the adapter tries to access IBTDTCCommitConfirm object as a part of the BatchComplete implementation, there is an access violation. Use the following solution to avoid this condition.

The problem is solved with an event in the following example. Here the interface pointer is accessed through a property that uses the event. The get always waits for the set.

protected IBTDTCCommitConfirm CommitConfirm
this.commitConfirm = value;
return this.commitConfirm;
protected IBTDTCCommitConfirm commitConfirm = null;
private ManualResetEvent commitConfirmEvent = new ManualResetEvent(false);

Now assign the return value from IBTTransportBatch::Done() to this property and use it in the BatchComplete call.

BizTalk Server does not handle all failures; the adapter must be configured to handle errors like "no subscription."

When an adapter submits an operation (or batch of operations) to BizTalk Server there can be various reasons for failure. The two most significant are:

  • The receive pipeline failed.
  • A routing failure occurred while publishing a message.

The messaging engine automatically tries to suspend the message when it gets a receive pipeline failure. The suspend operation may not always be successful due to the following issues.

If the messaging engine hits a routing failure while publishing a message, then the messaging engine will not even try to suspend the message.

It is always possible that a message will fail. In such a situation, the adapter should explicitly call the MoveToSuspendQ() API and should try to suspend the message. When an adapter tries to suspend a message, one of the following should be true:

  • The same message object that it had submitted (recommended) should be suspended.
  • If the adapter has to create a new message, then it should set the message context of the new message with the pointer to the message context of the message that was originally submitted. The reason for this is that the message context of a message has a lot of valuable information about the message and the failure. This information is required to debug the failed message.
If the adapter creates a new message object and suspends it, the adapter should copy the error information from the old message object to the new message object.

Some adapters, such as the HTTP adapter provided with BizTalk Server, do not require that the message be suspended. These adapters can just return an error back to their client.

Item 1: Causes of Failure

A simple cause of failure are the errors that can occur as the batch is constructed or when IBTTransportBatch::Done() is called.

  • Submit failure: The Submit call can fail for a limited number of reasons, and all of them are fatal. These reasons include:
    • Out-of-memory.
    • The schema assembly has been dropped from the deployment. In this case, the Submit fails with a cryptic error. In the MQSeries adapter, the generic failure exception from BizTalk Server is caught, and an extended error message is written in the system event log. This message suggests that one of the possible causes of the error is that the schema assembly has somehow been dropped from the deployment.
    • In general, if submit fails you should try to suspend the message using the same transaction.
  • IBTTransportBatch::Done() failure: The IBTTransportBatch::Done() call can fail for one of several reasons. In general, you should always attempt one suspend operation and only abort if that fails. One of the error codes you might receive from the failure of IBTTransportBatch::Done() is that BizTalk Server is trying to shut down. In this case, you should just end the transaction and leave it because the Terminate call is probably happening concurrently. Other scenarios occur when you have successfully constructed the batch and successfully executed IBTTransportBatch::Done(). In these cases, the errors are returned in BatchComplete and the adapter must work out what to do with them. The rest of the topic "Handling receive specific batch errors" deals with this case.

Item 2: BatchComplete Errors

BatchCompelete() is a callback provided by the adapter that is invoked by BizTalk Server to indicate the completion status of a batch operation.

The most important parameter passed to BatchComplete is the batch status hResult. This indicates success or failure for the batch. If the batch failed, it means that none of the operations in the batch have succeeded. (See the topic, "Batches are Atomic.") The adapter goes through the batch status structure and determines which messages failed (this is known as filtering the batch).

  • Non-Transactional BatchComplete Errors ― For non-transactional adapters, you must choose your response if a failure occurs for a SubmitMessage()/ SubmitRequestMessage() or SubmitResponseMessage() operation. Typically adapters suspend the message by calling MoveToSuspendQ().
    The following operations are always expected to pass: DeleteMessage(), MoveToSuspendq(), ResubmitMessage(). If these operations fail, it typically means that there is a bug in the adapter. You don't have to write code to handle a failure in these cases. However if the batch failed because another operation failed, then these operations must be re-executed in a fresh batch.
    If the adapter had called MovetoBackupTransport() and if that fails (because there was no backup transport), then the adapter should call MoveToSuspendq() to suspend the message
  • Transactional BatchComplete Errors ― When you submit batches to BizTalk Server using a transaction created by the adapter, you should follow one of these two scenarios:
    • Use Single message batches: Send a single-message batch to BizTalk Server. If that single message fails, then you can legally send BizTalk Server a second batch under this same transaction, this time making sure to move the offending message to the Suspend Queue rather than submitting it. This second batch should succeed and you can then commit the transaction, of course waiting for BizTalk Server to confirm that that second batch was successful. If it wasn’t, the adapter must end the transaction, or find somewhere else to place that message. In this scenario, you immediately take a significant performance hit. (Particularly so because you are using transactions.)
      There are some techniques that can still be used rescue the performance of the adapter. For example, the MQSeries adapter adapts its approach dynamically at runtime. It runs with 100 message batches. If it hits an error, it must end the batch, but it switches to single message batches for a short while as it eats past the bad message. It then lets the throttle out and reverts to 100 message batches. Of course, if it hits the error again, it again slows down.
      Unfortunately with this adapter, nothing could be done to stop the pipeline from seeing a failed message as a submit operation; an unavoidable consequence of this approach. Pipeline components (such as BAM pipeline reporting) see the message on the submit even though the message might actually be suspended.
    • Use preemptive suspension: Construct a multi-message batch in which the erroneous messages are preemptively suspended. The batch contains a mix of Submits and MoveToSuspendQ, and it will be the first and only batch under the transaction. It should succeed, given that the bad data was preemptively suspended, and the transaction can be committed (after waiting to receive the confirmation from BizTalk Server.)
      This would seem to require looking into the future, but this technique has actually been used in the MSMQ adapter. The trick is having reliable, unique message IDs. This adapter constructs a batch of messages; if anything fails it rolls back the transaction (and so the batch), but remembers the message ID in a temporary data structure. (In order to protect this structure from growing indefinitely, items in it are removed after some fixed time delay.) Before each batch is submitted, the adapter checks the list of bad message IDs. If it sees one, it knows that that message will fail (because it failed once in the past) and preemptively suspends it rather than trying to submit it.
      Not every adapter has a reliably unique message ID. (And if it’s a transactional store, it is less likely to have one). Because of this, many transactional adapters are restricted to sending single-message batches.

Item 3: Other Errors

In all other cases (such as failures in suspending messages), the adapter must end the transaction. Any other outcome results in either duplicate or dropped messages.

Whenever the adapter can, it should abort the transaction if a batch fails. However there are scenarios, where the adapter cannot abort the transaction. In such a scenario it should suspend the message using the same transaction.

Item 4: Transactional Receive Errors

A common transactional processing pattern is to end a transaction when an error occurs: everything returns to its previous state and no data is lost. However, if you are consuming data from some kind of transactional feed, for example, pulling a row at a time from some staging table on a database, or pulling one message at a time from a queuing product like MQSeries or MSMQ, then this might not be enough. If you simply end the transaction and go back and pick up the same data again, the same error is likely to occur and the system becomes stuck in a repeated loop.

The SQL adapter in BizTalk Server 2004 actually shipped with this behavior. The problem was that the adapter would see an error from BizTalk Server and end the transaction. However, soon after release the adapter was changed to attempt to suspend a failed message and commit the transaction. Moving a message to the suspend queue under the same transaction and then committing the transaction saves the data from ever being lost and also allows the adapter to eat past bad data.

When the receive portion of an adapter is passed an error message in response to a Submit message operation, the adapter should process that error and move the message to the Suspend Queue.

In the case of transactional batches in which the adapter has created the transaction object and submits messages under the transaction, the adapter should logically move the message to the Suspend Queue under the same transaction when failures occur. It is the transaction that ensures that data is not dropped. (And even data that is causing an error should never be dropped.)

Item 5: Suspending Messages on Failure

  • Routing Failures ― BizTalk Server does not accept a message to be published in its MessageBox if there are no subscriptions defined to accept it. Subscriptions are either orchestrations or send ports. Multiple subscriptions can be defined, in which case the message is sent to multiple destinations. If there are no subscriptions BizTalk Server rejects the message outright, and it will not attempt to suspend it. If the adapter does not handle this error and explicitly suspends the message, then the message is dropped and its data potentially lost. Of course a transactional adapter may end the transaction and return the message to its destination.
  • Pipeline Failures ― The receive side stream must support the Seek() method for BizTalk Server to be able to suspend the message on a pipeline failure. If the message stream is not seekable, then BizTalk Server will still try to run Seek(), and will generate an error.
    In many cases supporting Seek is not easy. When streaming data from a network for example, it may be difficult to go back to the network resource and request the data again.
    The trick that a number of adapters that shipped with BizTalk Server 2004 employed is to spool the message data onto disk at the same time as BizTalk Server reads the data. If BizTalk Server hits an error (in the pipeline processing of the message data for example), then it can use Seek() on the stream and the data will be read from the disk. Internally it uses the ReadOnlySeekableStream class that wraps an incoming non-seekable stream and overflows to disk when a configurable threshold is reached. For messages smaller than the threshold size, the disk is never hit, for large messages memory bloat is avoided.

Item 6: Customizing Adapter Error Handling

Sometimes there is no one correct response to an error. In this case, you should consider a user-configurable option to choose between behaviors. The MQSeries adapter does this.

The problem with having the adapter suspend messages when it sees an error is that the suspend queue in BizTalk Server 2004 is something of a “black hole." It is relatively easy to get messages in the queue, but a little tricky to get them out again.

Some users of the adapter might not want anything in the suspend queue. For example, in the case of the MQSeries adapter, the user is offered a configuration option to do one of the following:

  • Set the adapter to end the current transaction and disable itself when it sees an error.
  • Suspend the failed message and commit the transaction. The adapter does this even when BizTalk Server has successfully suspended the message. This action meets the requirements of the customer even if it causes the event log to not be strictly correct.

Item 7: Implementing Receive Ordering

The interface to BizTalk Server is designed for performance and the ability to scale out by supporting concurrency. However, if you want a strictly ordered receive of messages (as is sometimes required when receiving messages from a message queue product like MQSeries or MSMQ) then you must do some additional work in the adapter to disable some of that concurrency. This can be done in two steps:

  1. You must use a single thread for all the data processing in the adapter.
  2. You must wait for BizTalk Server to have completely processed each batch. This requirement is important and can be accomplished by using .NET thread synchronization primitives. For example, using an AutoResetEvent, you would:
    • Declare the event object where it can be accessed by both the main worker thread and the BatchComplete callback object.
    • On the main worker thread submit the messages to the batch as usual but then call AutoResetEvent.Reset() on the event object just before the call to the batch IBTTransportBatch::Done().
    • Call AutoResetEvent.WaitOne() on the event object from this same thread. This would cause the main worker thread to block. In the BatchComplete callback from BizTalk Server you would then call AutoResetEvent.Set() on the same event object to unblock the worker thread so it is ready to process another message.

It is strongly suggested that receive ordering like this be made configurable because it causes significant performance degradation. Many, if not most, user scenarios don’t actually require ordering of messages. Suspending messages can also break ordering. Exactly what to do in this case is really application-dependent, so the best thing for your adapter to do is to offer the user a configuration point.

In ordered scenarios, some customers have stated that they would rather stop the processing, that is, disable the adapter, rather than break ordering. The MQSeries adapter, which supports ordered receive provides this option to the user.

Here is a typical example of send-side batching:

  1. BizTalk Server gives a batch of messages to the adapter.
  2. When the adapter has determined that it has given the message to its destination correctly, it executes delete back on BizTalk Server indicating that it is done. (As usual, several delete messages can be arbitrarily batched up to improve performance.)

If the send side adapter fails to process a message then it may do one of several things with that message:

  • The adapter should tell BizTalk Server that it wants a message retried. BizTalk Server does not automatically retry a message. BizTalk Server keeps a count of the retries and this count can be seen in the message context.
  • An adapter may determine that it can't process a message. In this case, the adapter might move it to the next transport. The adapter does this with the MoveToNextTransport() call on the Batch object.
  • The adapter is also free to move the message to the Suspend Queue.

The adapter determines what happens to the message. However, it is recommended that you have adapters behave in a consistent manner because this makes a BizTalk Server installation easier to support.

It is highly recommended that adapters behave in the following way, as the adapters that are shipped with BizTalk Server do.

Item 1: Handling Send Errors

  1. The send adapter receives some messages and it submits them to BizTalk Server.
  2. For each successful message the adapter should delete that message on BizTalk Server. All communication back to BizTalk Server is done through batches and the deletes can be batched up. They do not have to be the same batch that BizTalk Server created on the adapter. If there are any response messages (as in a SolicitResponse scenario), then they should be submitted back to BizTalk Server (with SubmitResponse) along with the associated delete.
  3. If the message processing in the adapter was unsuccessful, check the retry count.
    1. If the retry count was not exceeded, resubmit the message to BizTalk Server remembering to set the retry time on the message. The message context provides the retry count and the retry interval the adapter should use.
    2. If the retry count has been exceeded , then the adapter should attempt to move the message using MoveToNextTransport.The resubmit and MoveToNextTransport messages can be mixed altogether along with the deletes in the same batch back to BizTalk Server. This is not required, but can be a useful step.
  4. The resubmit and the MoveToNextTransport are ways for the adapter to deal with failures. But there can be a failure within the processing of the failure. In this case, in processing the response from BizTalk Server (this is done in the BatchComplete method) the adapter must create another batch against BizTalk Server to indicate what to do with that failure. Follow these steps when processing a failure that occurs within the processing of another failure:
    1. If the resubmit fails, use MoveToNextTransport.
    2. If the MoveToNextTransport fails, use MoveToSuspendQ.

Ultimately we must keep creating batches on BizTalk Server until we have a successful action back on BizTalk Server.

Item 2: Separate Batches for Response Messages

When using transactions on the send side, the transaction created by BizTalk Server is used against the target system and then used for the delete back on BizTalk Server. If anything fails, the transaction can be ended in which case the delete is ended, and the data remains in BizTalk Server and not in the target system. But the adapter does not just end the transaction; it must also handle the state of the messages it was given correctly. Specifically, the adapter should call resubmit , MoveToNextTransport, and MoveToSuspendQ appropriately.

This is why, to make use of transactions on the send side, you must use the asynchronous batching API. Transactions are only supported for asynchronous send adapters. You should not try to use transactions with synchronous send adapters

It is very important to place the delete opperations and SubmitResponse opperations in a batch together that uses the transaction. Failure is handled by ending the transaction (to ensure that data is only submitted once to a external system), but you still want to resubmit or run MoveToNextTransport for the message back on BizTalk Server. The trick is to use a separate normal (non-transactional) batch for these types of operations.

The following figure shows the use of separate batches for response messages.

Figure 1 Using a separate batch for response messages

Using a seperate batch for response messages

Item 3: Sorting the Send Side Transactional Batches by Endpoint

Batches of messages sent by BizTalk Server to the adapter can span multiple send ports (or endpoints). Because the adapter typically only wants to have a transaction to a single endpoint, the adapter must sort the messages based on send port (SPName or OutboundTransportLocation). This way, the adapter can create a transaction that only spans a particular send port.

For example, when an FTP send adapter receives a batch of messages from BizTalk Server, it gets a mixed batch of messages for all the currently active FTP send ports. This happens because the API is singleton based, meaning that there is only a single FTP adapter loaded, not one per send port.

This leaves the adapter with some work to do. The adapter must first sort the batch of messages it was given by BizTalk Server into separate batches, one for each endpoint. After it has done that it can deal with each endpoint in turn and will probably construct delete batches for each endpoint. The BaseAdapter generic reusable classes in the SDK sample code works in the same way.

  • Dynamic Send ― A BizTalk Server orchestration can send a message to a port that has not been configured as long as it provides sufficient configuration details in the message header and in the URL itself. The protocol of the URL must be recognized by BizTalk Server.
  • Sorting for Dynamic Send ― When sorting messages, you should take care to establish what defines an endpoint. This is especially true in the case of dynamic send. If it’s just the URI that defines the endpoint, then things are quite simple. However, in an FTP session the username logon details might be used by the FTP server to define the true endpoint. In this case, if the adapter logs in as a different account, it may be connected to a different directory.
    In some cases, the true endpoint is not known until you have run the Enterprise Single Sign-On (SSO) command ValidateAndRedeemTicket.
    In the case of MQSeries, the determination of whether to use transactions was made configurable. Given the architecture and the use of a remote COM+ object, it turned out that it was best to regard a transactional endpoint as distinct from a non-transactional endpoint.
    To summarize, the sorting of messages into their single endpoint batches is sometimes a non-trivial task and may involve such extra steps as considering the context values and even the result of a call to SSO.
  • Sorting for Static Send ― If the endpoint is a configured endpoint rather than a dynamic send, then there is a unique GUID on the message context called the static port ID (SPID) that can be used for sorting the endpoint. The following code can be used to retrieve it:
    string spid = (string)message.Context.Read("SPID", "http://schemas.microsoft.com/BizTalk/2003/system-properties");
    This should come as something of a relief when you consider the problems introduced by the XML Schema Definition (XSD)-based configuration framework. With this you have a property that might be part of the endpoint key buried inside XML in a single property on the context. This property might well co-exist as a context property in itself. You should understand that if you have a SPID on the context, you can use that. Otherwise you are doing a dynamic send and you need to construct an alternative key to sort the batch by.
    The following figure shows message sorting by endpoint.
    Figure 2 Sorting send-side batches by endpoint
    Sorting messages by endpoint

Item 4: Retry Counts

You should keep in mind that the retry count of a message is not aware of the success or failure of a batch.

On the send side, a batch of messages may fail because a few messages in the batch have failed. The adapter must make a determination for every message that it receives and in the failed batch scenario, you might assume that every message is resubmitted. However, if all the messages in a failing batch are resubmitted, the retry count (which is maintained by the BizTalk Server engine) is incorrectly incremented even for the nonfailing messages because they happen to be in the same batch as the bad messages. In this case, an adapter could reform the outbound batch and retry the good messages against the external system.

Item 5: Message IDs

BizTalk Server provides a number of different IDs for a message and you should be careful to use the correct ID at the correct time.

  • A MessageID is specific for an instance of a message in memory, which means that a retry of a message comes with a different MessageID.
  • A WorkID stays constant across message retries.
  • An InterchangeID also stays constant across retries but could be shared with other messages in the system. If you need an ID which will stay constant across message retries, do not use the Interchange ID, because it is common to every message that results from a disassembly in the receive pipeline. This ID is used if a target system depends upon duplicate ID detection to enforce once-only delivery. The Microsoft BizTalk Adapter v2.0 for mySAP makes use of the WorkID for this purpose.

Item 6: Message and Batch Processing

Messages are processed synchronously within a batch by BizTalk Server, but many batches may be processed concurrently. This may provide an opportunity for some optimization in the adapter. There may be a optimal batch size to look for in the application domain; for example, all the files on an FTP site (until some limit is hit). In the case of SAP you might process a single stream into a number of messages which are then submitted as a batch.

The adapter determines the messages it puts in the different batches. The batch used by an adapter to pass operations back to BizTalk Server does not need to exactly correspond to the list of messages that BizTalk Server gives to the adapter. In other words, when doing transactional sends you must split the resubmit, MoveToNextTransport, and MoveToSuspendQ actions into separate batches.

Many adapters will sort a batch of messages it has been given for multiple endpoints into separate list of messages for further processing.

The point is that there are no rules (besides those associated with the transaction) that are associated with message batching in BizTalk Server. Batching is simply an implementation-specific way to chunk messages into and out of BizTalk Server.

An adapter can batch up messages from multiple smaller batches that BizTalk Server has given the adapter into a larger batch for the response to BizTalk Server. This might be a significant optimization in transactional adapters that are dealing with large numbers of very small messages.

Typically BizTalk Server 2004 produces send-side batches of between 5 and 10 messages. If these were very small messages, an adapter might choose to batch up to 100 messages or more before it submitted a transactional batch of deletes back to BizTalk Server.

An optimization like this would not be easy to implement; you must make sure that messages never got stuck in the adapter memory, waiting endlessly for some threshold to be met.

The significance of this batching can be seen in the performance numbers for BizTalk Server for the high throughput adapters like MQSeries. In these adapters, messages are received at least twice as often as they are sent. By default the receive uses batches of 100 messages and the send just uses the BizTalk Server batch it was given.

Item 1: Cleaning Up COM Objects

Use Marshal.ReleaseComObject() to clean up COM objects, particularly IBTTransportBatch.

When writing managed code that uses COM objects, the Common Language Runtime (CLR) runtime generates proxy objects that hold the actual references to the COM object. The proxy objects are managed objects and subjected to the usual rules of garbage collection. A problem arises in that the garbage collector only sees memory that it allocated, and has no awareness of the COM object itself. With the proxy objects being small a situation could exist where a large COM object is left in memory simply because the CLR garbage collector is oblivious to it.

To avoid this problem, explicitly release the underlying COM object if you are certain you are done with it. This is done by calling Marshal.ReleaseComObject().

ReleaseComObject returns the number of remaining references and only lets go of the COM object when this returned value is zero. Often ReleaseComObject is called in a loop to insure that the object is released. After that is done, you should call SuppressFinalize on this object because there is nothing to finalize. One last step is to check whether this really is a COM object.

if (Marshal.IsComObject (batch))
While (0 <Marshal.ReleaseComObject(batch)
GC.SuppressFinalize (batch);

In the case of the adapters that shipped with BizTalk Server, the programmers found that explicitly releasing the IBTTransportBatch object returned from TransportProxy GetBatch can make a significant improvement to performance.

Item 2: Initializing and Closing an Adapter

In order for BizTalk Server to recognize your code as an adapter, you must implement an interface called IBTTransportControl. This interface defines how BizTalk Server communicates with your adapter, and is defined as follows:

public interface IBTTransportControl 
void Initialize(IBTTransportProxy transportProxy);
void Terminate();

The interface contains two methods, Initialize () and Terminate ():

  • Initialize() ― BizTalk Server calls the Initialize method after it loads the adapter assembly. It does this to pass the Transport Proxy (the main handle to BizTalk Server) to the adapter. The implementation of Initialize is straightforward; simply store the Transport Proxy in a member variable.
  • Terminate() ― Terminate is called by BizTalk Server on shutdown, to give the adapter time to finish the execution of all batches. This makes the implementation of the Terminate() method much more involved.
    The adapter should not return from a Terminate() call until any pending work it has is complete. When BizTalk Server calls Terminate, the adapter should try to stop all its current tasks and not start any new ones.
    Because the Terminate call is called as part of the service shutdown, the service control manager ends the process if the adapter perpetually blocks in Terminate. In this case, you see the warning from the service control manager as it kills the BizTalk Server service. This should, of course, be avoided. If the adapter does not handle this appropriately, and still has threads running when the process starts to shut down, then you will occasionally see an access violation from BizTalk Server on shutdown.

Because of the asynchronous nature of the interface to BizTalk Server, it is likely that under load there will be many batches and therefore threads still being executed. The Terminate call should be implemented to wait on the conclusion of every batch the adapter has successfully executed on BizTalk Server. The conclusion of the batch is signaled by the batchComplete callback from BizTalk Server. So the Terminate call should wait on every pending batchComplete to happen. However, the execution of the batch must be successful; that is, the call to IBTTransportBatch::Done() must not have failed. If the call to IBTTransportBatch::Done() fails, there is no batch Callback.

After you realize that you have to add synchronization code to your adapter, the implementation is fairly straightforward.

One simple approach is to implement a compound synchronization object with enter() and leave() methods for the worker threads and a terminate() method that blocks while a thread is still within the protected execution. (Incidentally, the solution is very similar to the familiar multiple reader, single writer structure where the worker threads can be thought of as readers and the terminate method as the writer.)

The terminate method is as follows:

void Terminate ()

And for each worker thread:

If (!this.control.Enter())
return; // we can’t enter because Terminate has been called
//  create and fill batch
catch (Exception)
//  we are not expecting a callback

And finally in the callback from BizTalk Server:

batchComplete (…)
//  the callback from BizTalk Server
//  process results

BizTalk Server 2004 shipped with sample code, ControlledTermination, for the synchronization object described here.

Item 1: Receiving and Sending Asynchronously

The BizTalk Server messaging APIs have rich support for asynchronous programming. If you want to write a scalable adapter, plan on using the asynchronous model from the start because the asynchronous model provides better concurrency.

On the receive side, when an adapter submits a batch of messages to the BizTalk Server messaging engine (by calling IBTTransportBatch::Done()), the messaging engine queues up the work using its internal thread pool and returns immediately. The messaging engine processes the messages on a separate thread, leaving the adapter free to read more messages from its source and submit them without waiting for the previous message processing to complete.

On the send side your adapter can be either asynchronous or synchronous. However, if your protocol supports asynchronous operations, you should use this support to write a scalable adapter. For example, FILE and HTTP send adapters are fully asynchronous and they perform very few blocking/synchronous operations.

Asynchronous operations ensure that both the BizTalk Server messaging engine and your adapter will continue to do their respective work in parallel and not wait on each other for normal message processing.

Item 2: Batching to Improve Performance

Batching is the best starting point for writing a scalable adapter. This is true for both send-side and receive-side adapters. As explained in the Batching section of this document, every batch goes through a database transaction within BizTalk Server even if your adapter is non-transactional. Because there is a fixed delay associated with each transaction, you should try to minimize the number of these transactions by combining more than one operation into a single batch.

Item 3: Don't Starve the .NET Thread Pool

Writing BizTalk Server adapters is an exercise in writing .NET runtime code; writing .NET runtime code is invariably an exercise in asynchronous programming.

While starving the .NET thread pool is a risk to all asynchronous programming in .NET, it is particularly important for the BizTalk Server adapter programmer to watch out for this. It has impacted many BizTalk Server adapters: take great care not to starve the .NET thread pool.

The .NET thread pool is a limited but widely shared resource. It is very easy to write code that uses one of its threads and holds onto it for ages and in so doing blocks other work items from ever being executed.

Whenever you use BeginInvoke or use a Timer, you are using a .NET thread pool thread. If you have multiple pieces of work to do (for example copying messages out of MQSeries into BizTalk Server), you should execute one work item (one batch of messages into BizTalk Server) and simply requeue in the thread pool if there is more work to do. What ever you do, don't sit in a while loop on the thread.

In concrete terms this means replacing while loops with repeated calls to BeginInvoke.

This simple change will dramatically improve the responsiveness and ability to scale out for the whole implementation. It is possible to have a mainframe dump a million messages on an MQSeries queue ready to be consumed by BizTalk Server: in that situation you don’t want to be sitting on a while loop eating through those messages.

Item 4: Limiting Batch Size

If you are submitting messages to BizTalk Server in batches, don’t limit the batch just based on the message count. Consider what happens when an adapter has been configured to batch based on just message count: If the batch size is two and the adapter gets four messages of size 4KB, 8KB, 1MB and 5MB respectively, the first batch will be of size 12KB, and second batch will be of size 6MB.

Because the BizTalk Server messaging engine processes all messages in a single batch sequentially, the second batch will be processed much more slowly than the first batch. This is effectively reducing throughput. A better way to handle this problem is to batch based on both message count and total bytes in the batch (that is, batch size in bytes). There is no magic number for total bytes. However, in a normal processing scenario, if the batch size exceeds 1MB, you will start seeing poor concurrency and throughput.

Generally adapters are message agnostic and they do not know what the size of the messages will be in the production environment. The sizes of the incoming messages are likely to vary quite significantly. Because of this, always use message count and total bytes to build the batch.

Item 4: Throttling on the Send-Side

If you are writing a send adapter for a protocol that is inherently slow in nature (such as HTTP, FTP or two-way SOAP), consider the following:

  • Such an adapter might receive messages for transmission from the BizTalk Server messaging engine faster than it can transmit them. This discrepancy cause problems at various levels. The messages under transmission remain in memory and take up the virtual memory, slowing down the entire system.
  • The adapter might take up protocol-specific resources. For example, it might open too many concurrent connections to the server, which could disrupt the remote server.
  • The adapter might affect other adapters. For example, if too many messages queue up for a particular adapter, the BizTalk Server messaging engine will stop issuing requests to other send adapters in that process.

A solution is to put the slow and fast adapters in separate BizTalk Server hosts and control the number of messages using the "High Watermark" and "Low Watermark" settings.

Item 5: Throttling on the Receive-Side

Why would a customer want to slow down a receive adapter? There are numerous situations in which a receive adapter receives messages faster than the rate at which the rest of the system can process the messages. When such a situation occurs, the MessageBox database becomes backlogged. When this happens, the performance of the whole system drops dramatically.

If this is happening with your adapter, you can use one of the following techniques to reduce the speed of the receive adapter.

  1. Reduce the messaging engine thread pool size. A user can control the number of threads used by the messaging engine to publish messages into the MessageBox. By reducing the number of threads used by the messaging engine we reduce the rate at which the receive adapter will receive messages into the MessageBox. This setting only needs to be done for the host corresponding to the receive handler for the adapter. You should not set this for the host corresponding to the send handler for the adapter, unless you want to slow down the send adapter as well.
  2. Reduce the adapter batch size. Most fast receive adapters publish messages to the MessageBox in batches. The size of these batches is usually configurable in the receive location property page. By decreasing the batch size we can decrease the overall throughput of messages coming into the system.
  3. Change other adapter specific settings. After you complete the two previous steps, you can try adjusting other adapter parameters to further decrease throughput. Some adapters expose internal parameters that can be used to decrease throughput. For example, MQSeries adapter has a setting for “Ordered Delivery.” Ordered Delivery specifies that the adapter will publish a batch of messages, wait for it to complete, and then publish the next batch. By enabling this setting, you essentially remove all parallelism from the receive adapter. Conversely, tuning the parameters in the opposite way can be used to increase the receiving rate of a receive adapter.

Item 1: Adapter Installation Checklist

A complete adapter installation process consists at least following steps:

  1. Check dependencies. For example, the MSMQ adapter installer checks for the existence of the local MSMQ Service because the adapter runtime depends on it.
  2. Setup the local file system. This includes creation of installation folders and copying binary and support files.
  3. Put managed assemblies into the system global assembly cache.
  4. Create registry keys for the adapter.
  5. Add the adapter to BizTalk Server. This means to make new entries in the BizTalk management database for the adapter as well as for the adapter context property schema. This step is sometimes done manually using the BizTalk Server Administration plug-in for the Microsoft Management Console (MMC).

Item 2: Exceptions

The adapter installer may not need to check the dependencies in partial BizTalk Server installations. For example, in the BizTalk Server Administration only installation, the adapter installer does not need to check adapter runtime dependencies because the adapter runtime is not used. The same is true in the BizTalk Server runtime only installation, where the adapter installer does not need to check the adapter design time dependencies.

In multiple BizTalk Server environments, step 5 in the list immediately above (Add the adapter to BizTalk Server) only happens in the adapter installation on the first BizTalk Server, because all BizTalk Server service instances share the same BizTalk management database (with the default name, BizTalkMgmtDB). If step 5 is performed more than once in these environments, the additional steps fail. An adapter installer should keep this in mind.

Item 3: Installing the Adapter Assemblies into the Global Assembly Cache (GAC)

To support multiple BizTalk Server host environment with different adapter installation paths, the adapter assemblies must be installed into system global assembly cache.

During the adapter installation and configuration, a new entry is created in the adm_adapter table of the BizTalk management database (with the default name BizTalkMgmtDB) for this newly added adapter. This entry contains all reference information used by BizTalk Server for both runtime and design time.

Two fields, InBoundAssemblyPath and OutboundAssemblyPath are used by BizTalk Server runtime to find out where to load the adapter binaries. Because there is only one BizTalk Management database for a BizTalk Server group, all BizTalk servers in the group must share this same piece of information. If you want to install the adapter on different BizTalk Servers within the group, you must use the same installation path on each of those servers. If the local installation path is different from the path stored in the BizTalk Configuration database, BizTalk Server will not be able to load the adapter binaries.

Therefore, always sign the adapter with a strong name and use Gacutil.exe to put the adapter assembly into system global assembly cache during the adapter installation. Make sure that you leave the two fields InBoundAssemblyPath and OutboundAssemblyPath null, or empty.

BizTalk Server 2004 introduced a mechanism for generating property sheets for the adapter configuration user interface based on an XML Schema Definition (XSD) definition of the configuration properties. The use of this framework introduces a number of significant challenges for the adapter writer. This section covers the various problems this technology introduced and suggests effective workarounds.

Item 1: Custom Property Editors

It is impossible to put significant property validation on top of properties in the property sheet. The framework attempts to use XML Schema Definition (XSD) simpleType restrictions to define the validation rules for the UI. The simple rules for maximum and minimum value are applied but the regular expression restriction for string fields is not supported.

Also, the data is converted into CLR types before the restriction is applied so the user of the UI can sometime get a cryptic type conversion error rather than an out of range error. All in all, the validation logic is very weak.

The solution many adapter writers have adopted is to write custom property editors for the main properties on their property sheet. If there are a number of cross dependent properties (as is often the case), then the custom property editor can mangle the result into a single field and the runtime and then un-mangle it. This is the only way to get the necessary (though still generally trivial) custom code into the interface.

Custom Property editors are relatively straightforward to write and the property in the XSD can be annotated to use them. The annotation references an assembly which exposed the property editor entry point, and it is coded to show a CLR Form. You can now write a regular user interface and use the traditional interface generation tools that Visual Studio has to offer.

The following code shows how to use the XSD to add a custom property editor:

<xs:element name="queueDetails" type="xs:string" minOccurs="0">
               <baf:displayname _locID="queueName">Queue Definition</baf:displayname>
               <baf:description _locID="receiveQueueDesc">Details of MQSeries Server, Queue Manager and Queue from where you want to receive messages.</baf:description>
               <baf:category _locID="mqsCategory">MQSeries</baf:category>
               <baf:editor assembly="Microsoft.BizTalk Server.Adapter.MQSAdmin.dll">Microsoft.BizTalk Server.Adapter.MQSAdmin.MQSUITypeEditor</baf:editor>

Where the code referenced should look like this:

public class MQSUITypeEditor : System.Drawing.Design.UITypeEditor
public override System.Drawing.Design.UITypeEditorEditStyle GetEditStyle
(System.ComponentModel.ITypeDescriptorContext context) 
return System.Drawing.Design.UITypeEditorEditStyle.Modal;
public override object EditValue (System.ComponentModel.ITypeDescriptorContext context,
System.IServiceProvider provider, object value) 
Form qdForm = new QueueDefinitionForm(value);
IWindowsFormsEditorService service = 
DialogResult dialogResult = service.ShowDialog(qdForm);
class QueueDefinitionForm : System.Windows.Forms.Form 
//  traditional UI code goes here!

The administration runtime must be able to find the assembly referenced in the XSD, so you should add it to the GAC.

Item 2: Dynamic Send Issues

In the case of a dynamic send port in an orchestration, the message engine invokes an appropriate adapter based on the prefix of the URL string (as set through myDynamicPort(Microsoft.XLANGs.BaseTypes.Address) = "<URL>).

In the orchestration, if the URL string is “SQL://…..”, the SQL adapter is invoked, if the URL is “MSMQ://……”, then the MSMQ adapter is invoked. This prefix is defined in the adm_AdapterAlias table in the BizTalk Configuration database. After the adapter send runtime is invoked, it receives the URL string from the OutboundTransportLocation property of the SystemMessageContext object.

The part of the URL string after the prefix contains information for the adapter runtime to perform the send operation. For example, in the case of the SQL adapter, if the URL string is SQL://MyDatabaseServer/MyDatabase, the SQL adapter send runtime knows the data that is going to the database MyDatabase on the server MyDatabaseServer.

In many cases the simple URL string cannot carry all the information that the send adapter needs. In the case of the SQL adapter, the send adapter needs to also know the message RootElementName and TargetNamespace. For the additional information, there are two solutions here:

  • Put the information into send handler properties. This approach has an obvious limitation: the global availability defeats the dynamic purpose.
  • Implement adapter-specific message context properties.

Item 3: Avoiding Duplicate Properties

You must take care in writing the runtime code because of an issue introduced by XSD-generated property sheets. The runtime is often presented with the exact same property in a number of places: both property sheet Document Object Models (DOMs), the message context and even the location URI. As with any de-normalized model there is a potential for confusion. Adapters should first consider the handler properties, then override them with location properties, and then finally override them with values taken from the BizTalk Server message context and URI.

Item 4: Uniform Resource Identifier (URI)

An adapter must provide a Uniform Resource Identifier (URI) on its location property sheet but should make this URI hidden if possible. You must identify every BizTalk Server endpoint location with a URI and the XML Schema Definition (XSD) for the location should include this URI as a field. However, in general it is best to derive the URI from other fields rather than have the user enter it directly.

There is usually a simple way to do this. For example the FILE and FTP adapters generate URIs like: file://c:/myFolder/*.txt or ftp://myServer:8008/myFolder/*.xml. In these cases, the URI should be hidden in the property sheet. This is done by marking browsable show=false in the XSD.

For example:

<xs:element name=”uri” type=”xs:string”>
<baf:designer xmlns:baf=”BizTalk ServerAdapterFramework.xsd”>
<baf:browsable show=”false” />

You should generate the URI and add it to the Document Object Model (DOM). When this is done correctly, the location URI appears in the parent property page. In fact the whole location property sheet is opened as an extension of editing this URI property.

The URI shown in this parent property page is generally not escaped, that is, illegal URI characters, like a space, are not encoded. This is just a convention and not of particular significance. To BizTalk Server these are just strings. Using the Common Language Runtime (CLR) URI class to manipulate URIs in code is generally a good idea, but you should watch for exceptions. For example, in MQSeries, queue manager and queue names may contain a forward slash, which must be explicitly handled.

The point is that URIs have their own syntax and escaping rules and these can matter occasionally in the adapter code but in general does not affect BizTalk Server.

Item 5: Localization Issues

By using the Adapter Framework, adapter developers can implement adapter property pages with XML Schema Definition (XSD) schemas.

If your adapter has no globalization or localization requirement, then you can hardcode the XSD schema string inside the IDynamicAdapterConfig:GetConfigSchema() function.

If your adapter has globalization or localization requirements, you can implement the XSD schema in one of two ways.

  • Use separate XSD files outside the design-time binary. Make the whole text of the schema a manifest resource.
  • Dynamically replace the Property Names and Description from the resource:
    • Add a _locID to each element that you want to localize.
    • Use an xpath to pull back all the nodes in the schema that have a _locID attribute.
    • Look up the resources for a string indexed by the value of the _locID.
    • Replace the node text with the result.

The following is sample code for the second option:

string mySchema = GetSchemaFromResource(“mySchema”);
string myLocalizedSchema = LocalizeSchemaDOM (mySchema, resourceManager);
//  where…
protected string GetSchemaFromResource (string name)
Assembly assem = this.GetType().Assembly;
Stream stream = assem.GetManifestResourceStream(name);
StreamReader reader = new StreamReader(stream);
string schema = reader.ReadToEnd();
return schema;

protected XmlDocument LocalizeSchemaDOM (string schema, ResourceManager resourceManager)
XmlDocument document = new XmlDocument();
XmlNodeList nodes = document.SelectNodes
foreach (XmlNode node in nodes)
string locID = node.Attributes["_locID"].Value;
node.InnerText = resourceManager.GetString(locID);
return document;

Item 6: Managing Passwords

Consider using BizTalk Server Enterprise Single Sign-On (SSO) to handle credentials. If you put credentials directly in the properties of an endpoint, the password field will be blanked out when you need to export a binding file. This will require your user to re-key in the password as an administrator. You can avoid this difficulty by using SSO for credentials.

If the adapter endpoint has a Password property, be aware that the actual value is stored in clear text in the SSO Configure Store database despite the fact that it is displayed in the UI as "*". This property is also transferred through the network and a simple script using the BizTalk Server sample ExplorerOM will be able to read it.

Item 7: Handler Properties as Strings

Handler properties should be strings if used as default configurations.

Although it seems attractive to use the properties on the XSD-generated handler property sheet as defaults for their Location properties, there are several issues that make this less useful.

This seems attractive because if the value is not set in the Location the runtime automatically uses the value set in the handler.

The problem comes with knowing whether the value presented to the runtime is to be overridden or not. The typical way of doing this would be to have some notion of NULL defined for values and then a test could be made against that value. The problem when using the XSD based property sheets in BizTalk Server is that NULL is only supported for strings.

Even if you want your adapter to have default settings through the use of this NULL test and are willing to restrict the adapter to string types, it is still exposed to a very odd piece of user interface.

The XSD-generated property sheets only support the setting of a property back to NULL by right clicking on the property at which point a nullify? context menu appears and the property can be set to NULL. There is no visual feedback at all as to whether a property is NULL or not.

Ideally programmers like to code against strongly typed object models. Manipulating XML in code can at first seem very awkward and prone to error, but a couple of tricks and smart use of the support offered by the .NET Framework can dramatically simplify matters.

  • Do Not Create XML Documents with String Concatenation ― One of the worst mistakes to make with XML is to try and generate it from string concatenation and print statements in memory. Even for the most trivial XML snippet, it is easier to use a tool like XmlWriter or the DOM.
    If you are using XmlWriter, you should never be tempted to use the raw write capability because the writer immediately loses the state of the document, and you are left on your own: that is how to get incorrect XML out of an XmlWriter!
    At run time, the XmlWriter is preferred over the Xml DOM because of memory bloat issues associated with the DOM. However, at configuration or design time you may have more room for movement. Using the DOM facilitates the use of xpath queries which can be a very nice additional tool.
  • Consider Defining the Skeleton of Your XML Document as a Resource ― If you are generating a large XML document from a design tool and that generated document always follows the same basic structure, consider placing the whole skeletal XML file as a resource in the project. Now it is easy to edit it when you need to.
    Load the code into a DOM and then add the necessary flesh to the bones of the document by using xpath to pick out the node you want to add it to.
    This is exactly the technique used by the SQL adapter Schema Wizard that shipped with BizTalk Server 2004. In this case, you are creating a WSDL file, so the wizard stores the skeletal WSDL file in a resource and adds the generated XML Schema Definition (XSD) child parts by using selectNode with an xpath to find the right parent. This is user interface code so performance is not an issue; the resulting implementation is very clean, robust, and maintainable.

This section contains hints and tips that adapter developers have learned while designing adapters.

Item 1: Placing Processing Steps in the BizTalk Server Pipeline

In general the adapters built at Microsoft move message format-based processing out of the adapter and into the BizTalk Server pipeline. A good example is an adapter to a structured but non-XML data source.

In this case, it just gets the data and the BizTalk Server pipeline is used to parse it and convert it into an XML equivalent. The benefit is that the pipeline component itself becomes a reusable piece of the architecture.

Item 2: Making Adapter Behavior Configurable

One of the lessons learned from the MQSeries adapter beta program was that not all customers were happy with the same behavior, particularly when it came to how to handle errors and ordering.

The solution was to make the behavior configurable: you can specify whether the adapter is to support ordering, whether failures are moved to the suspend queue, or whether they cause the adapter to stop processing and disable itself.

Making such behaviors configurable can significantly simplify customers' lives where they would otherwise have to write complex orchestrations or scripts external to BizTalk Server to achieve the same result.

Item 3: Supporting Correlation with Message Queues

Many messaging platforms support the notion of a correlation ID in the message header to support an application level request response scenario. Examples include MQSeries, MSMQ, and SQL Service Broker. It would seem attractive to map the request response pattern of the external messaging system to a send-response adapter in BizTalk Server, however this doesn't make sense because of where the transactions lie. Specifically, the send to the external messaging system requires a transactional commit before the other end of the queue sees the data. The receive must also a separate transaction.

The solution in BizTalk Server is to:

  • Use correlation sets in orchestration
  • Configure two separate ports: one for the send and one for the receive

In a simple case the orchestration specifies the correlation ID that will be associated with the message by the adapter (it would be passed to the adapter as a context property on the message). In a more complex case, the scenario calls for the external messaging system to allocate the ID; in this case it can be passed back from the send port to the orchestration with a response message. This response message is just to pass back the ID and not the true message response.

There is a race condition in the orchestration engine such that the true response to the message could win against the ID response from the send. This race condition must be handled in the orchestration itself.

Writing BizTalk Server adapters can be an involved process, with many issues to consider. We hope that the information found in this paper will help speed you on that path.

If you have comments and suggestions regarding future versions of this paper, please send them to btsdf@microsoft.com.