Optimizing Pipeline Performance

1. Because pipeline components have a significant impact on performance (for example, a pass-through pipeline component performs up to 30 percent better than an XML assembler/disassembler pipeline component), make sure that any custom pipeline components perform optimally before implementing them in your deployment. Minimize the number of pipeline components in your custom pipelines if you want to maximize the overall performance of your BizTalk application.
   
   
2. You also can improve overall performance by reducing the message persistence frequency in your pipeline component and by coding your component to minimize redundancy. Every custom assembly and in particular artifacts that could potentially disrupt performance, like custom tracking components, should be tested separately under heavy load condition to observe their behavior when the system is working at full capacity and to find any possible bottlenecks.
   
   
3. If you need to read the inbound message inside a pipeline component, avoid loading the entire document into memory using an XmlDocument object. The amount of space required by an instance of the XmlDocument class to load and create an in-memory representation of a XML document is up to 10 times the actual message size. In order to read a message, you should use an XmlTextReader object along with an instance of the following classes:
   
   
   • VirtualStream (Microsoft.BizTalk.Streaming.dll) - The source code for this class is located in two locations under the Pipelines SDK as follows: SDK\Samples\Pipelines\ArbitraryXPathPropertyHandler and SDK\Samples\Pipelines\SchemaResolverComponent\SchemaResolverFlatFileDasm.
     
     
   • ReadOnlySeekableStream (Microsoft.BizTalk.Streaming.dll).
     
     
   • SeekAbleReadOnlyStream - The source code for this class is located in two locations under the Pipelines SDK as follows: SDK\Samples\Pipelines\ArbitraryXPathPropertyHandler and SDK\Samples\Pipelines\SchemaResolverComponent\SchemaResolverFlatFileDasm.
     
     
4. Use the PassThruReceive and the PassThruTransmit standard pipelines whenever possible. They do not contain any pipeline component and do not perform any processing of the message. For this reason, they ensure maximum performance in receiving or sending messages. You can use a PassThruReceive pipeline on a receive location if you need to publish a binary document to the BizTalk MessageBox and a PassThruTransmit pipeline on a send port if you need to send out a binary message. You can also use the PassThruTransmit pipeline on a physical send port bound to an orchestration if the message has been formatted and is ready to be transmitted. You will need to use a different approach if you need to accomplish one of the following actions:
   
   
   • Promote properties on the context of an inbound XML or Flat File message.
     
     
   • Apply a map inside a receive location.
     
     
   • Apply a map in an orchestration that subscribes to a message.
     
     
   • Apply a map on a send port that subscribes to a message.
     
     
   To accomplish one of these actions, you must probe and discover the document type inside the receive pipeline and assign the (namespace#root-name) value to the MessageType context property. This operation is typically accomplished by a disassembler component such as the Xml Disassembler component (XmlDasmComp) or the Flat File disassembler component (FFDasmComp). In this case, you need to use a standard (for instance, XmlReceive pipeline) or a custom pipeline that contains a standard or a custom disassembler component.
   
   
5. Acquire resources as late as possible and release them as early as possible. For example, if you need to access data on a database, open the connection as late as possible and close it as soon as possible. Use the C# using statement to implicitly release disposable objects or the finally block of a try-catch-finally statement to explicitly dispose your objects. Instrument your source code to make your components simple to debug.
   
   
6. Eliminate any components from your pipelines that are not strictly required to speed up message processing.
   
   
7. Inside a receive pipeline, you should promote items to the message context only if you need them for message routing (Orchestrations, Send Ports) or demotion of message context properties (Send Ports).
   
   
8. If you need to include metadata with a message, and you don't use the metadata for routing or demotion purposes, use the IBaseMessageContext.Write method instead of the IBaseMessageContext.Promote method.
   
   
9. If you need to extract information from a message using an XPath expression, avoid loading the entire document into memory using an XmlDocument object just to use the SelectNodes or SelectSingleNode methods. Alternatively, use the techniques described in Optimizing Memory Usage with Streaming.
   
   

The following techniques describe how to minimize the memory footprint of a message when loading the message into a pipeline.

Use ReadOnlySeekableStream and VirtualStream to process a message from a pipeline component

int bufferSize = 0x280;
int thresholdSize = 0x100000;
Stream vStream = new VirtualStream(bufferSize, thresholdSize);
Stream seekStream = new ReadOnlySeekableStream(inboundStream, vStream, bufferSize);

Using XPathReader and XPathCollection to extract a given IBaseMessage object from within a custom pipeline component.

If specific values need to be pulled from an XML document, instead of using the SelectNodes and SelectSingleNode methods exposed by the XmlDocument class, use an instance of the XPathReader class provided by the Microsoft.BizTalk.XPathReader.dll assembly as illustrated in the following code example.

Note
For smaller messages especially, using an XmlDocument with SelectNodes or SelectSingleNode may provide better performance than using XPathReader, but XPathReader allows you to keep a flat memory profile for your application.

This example shows how to use the XPathReader and XPathCollection to extract a given IBaseMessage object from within a custom pipeline component.

Copy

public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
    try
    {
        ...
        IBaseMessageContext messageContext = message.Context;
        if (string.IsNullOrEmpty(xPath) && string.IsNullOrEmpty(propertyValue))
        {
            throw new ArgumentException(...);
        }
        IBaseMessagePart bodyPart = message.BodyPart;
        Stream inboundStream = bodyPart.GetOriginalDataStream();
        VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
        ReadOnlySeekableStream readOnlySeekableStream = new ReadOnlySeekableStream(inboundStream, virtualStream, bufferSize);
        XmlTextReader xmlTextReader = new XmlTextReader(readOnlySeekableStream);
        XPathCollection xPathCollection = new XPathCollection();
        XPathReader xPathReader = new XPathReader(xmlTextReader, xPathCollection);
        xPathCollection.Add(xPath);
        bool ok = false;
        while (xPathReader.ReadUntilMatch())
        {
            if (xPathReader.Match(0) && !ok)
            {
                propertyValue = xPathReader.ReadString();
                messageContext.Promote(propertyName, propertyNamespace, propertyValue);
                ok = true;
            }
        }
        readOnlySeekableStream.Position = 0;
        bodyPart.Data = readOnlySeekableStream;
    }
    catch (Exception ex)
    {
        if (message != null)
        {
            message.SetErrorInfo(ex);
        }
        ...
        throw ex;
    }
    return message;
}

Another method for implementing a custom pipeline component which uses a streaming approach is to use the .NET XmlReader and XmlWriter classes in conjunction with the XmlTranslatorStream class provided by BizTalk Server. For example, the NamespaceTranslatorStream class contained in the Microsoft.BizTalk.Pipeline.Components assembly inherits from XmlTranslatorStream and can be used to replace an old namespace with a new one in the XML document contained in the stream. In order to use this functionality inside a custom a pipeline component, you can wrap the original data stream of the message body part with a new instance of the NamespaceTranslatorStream class and return the latter. In this way the inbound message is not read or processed inside the pipeline component, but only when the stream is read by a subsequent component in the same pipeline or is finally consumed by the Message Agent before publishing the document to the BizTalk Server MessageBox.

The following example illustrates how to use this functionality.

public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
   IBaseMessage outboundMessage = message;
   try
   {
      if (context == null)
      {
         throw new ArgumentException(Resources.ContextIsNullMessage);
      }
      if (message == null)
      {
         throw new ArgumentException(Resources.InboundMessageIsNullMessage);
      }

      IBaseMessagePart bodyPart = message.BodyPart;
      Stream stream = new NamespaceTranslatorStream(context, 
                                                    bodyPart.GetOriginalDataStream(), 
                                                    oldNamespace, 
                                                    newNamespace);
      context.ResourceTracker.AddResource(stream);
      bodyPart.Data = stream;
   }
   catch (Exception ex)
   {
      if (message != null)
      {
         message.SetErrorInfo(ex);
      }

      throw ex;
   }
   return outboundMessage;
}

The following information was taken from Nic Barden’s blog, http://blogs.objectsharp.com/cs/blogs/nbarden/archive/2008/04/14/developing-streaming-pipeline-components-part-1.aspx (http://go.microsoft.com/fwlink/?LinkId=160228). This table provides a summarized comparison of loading messages into pipelines using an in-memory approach and using a streaming approach.

Concepts