Optimizing BizTalk Server Performance
This topic provides recommendations for optimizing performance of the BizTalk Server computers used in a production BizTalk Server environment. The optimizations listed in this topic are applied after BizTalk Server has been installed and configured.
The following recommendations can be used to increase BizTalk Server performance:
Create multiple BizTalk Server hosts and separate host instances by functionality
Separate hosts should be created for sending, receiving, processing, and tracking functionality. Creating multiple BizTalk hosts provides flexibility when configuring the workload in your BizTalk group and is the primary means of distributing processing across the BizTalk Servers in a BizTalk group. Multiple hosts also allow you to stop one host without affecting other hosts. For example, you may want to stop sending messages to let them queue up in the MessageBox database, while still allowing the inbound receiving of messages to occur. Separating host instances by functionality also provides the following benefits:
- Each host instance has its own set of resources such as memory, handles, and threads in the .NET thread pool.
- Multiple BizTalk hosts will also reduce contention on the MessageBox database host queue tables because each host is assigned its own work queue tables in the MessageBox database.
- Throttling is implemented in BizTalk Server at the host-level. This allows you to set different throttling characteristics for each host.
- Security is implemented at the host-level; each host runs under a discrete Windows identity. This would allow you, for example, to give Host_A access to FileShare_B, while not allowing any of the other hosts to access the file share.
|While there are benefits to creating additional host instances, there are also potential drawbacks if too many host instances are created. Each host instance is a Windows service (BTSNTSvc.exe), which generates additional load against the MessageBox database and consumes computer resources (such as CPU, memory, threads).|
For more information about modifying BizTalk Server Host properties, see "How to Modify Host Properties" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkId=101588.
Configure a dedicated tracking host
BizTalk Server is optimized for throughput, so the main orchestration and messaging engines do not actually move messages directly to the BizTalk Tracking or BAM databases, as this would divert these engines from their primary job of executing business processes. Instead, BizTalk Server leaves the messages in the MessageBox database and marks them as requiring a move to the BizTalk Tracking database. A background process (the tracking host) then moves the messages to the BizTalk Tracking and BAM databases. Because tracking is a resource intensive operation, a separate host should be created that is dedicated to tracking, thereby minimizing the impact that tracking has on hosts dedicated to message processing.
Using a dedicated tracking host also allows you to stop other BizTalk hosts without interfering with BizTalk Server tracking. The movement of tracking data out of the MessageBox database is critical for a healthy BizTalk Server system. If the BizTalk Host responsible for moving tracking data in the BizTalk group is stopped, the Tracking Data Decode service will not run. The impact of this is as follows:
- HAT tracking data will not be moved from the MessageBox database to the BizTalk Tracking database.
- BAM tracking data will not be moved from the MessageBox database to the BAM Primary Import database.
- Because data is not moved, it cannot be deleted from the MessageBox database.
- When the Tracking Data Decode service is stopped, tracking interceptors will still fire and write tracking data to the MessageBox database. If the data is not moved, this will cause the MessageBox database to become bloated, which will impact performance over time. Even if custom properties are not tracked or BAM profiles are not set up, by default some data is tracked (such as pipeline receive / send events and orchestration events). If you do not want to run the Tracking Data Decode service, turn off all tracking so that no interceptors save data to the database. To disable global tracking, see "How to Turn Off Global Tracking" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkId=101589. Use the BizTalk Server Administration console to selectively disable tracking events.
The tracking host should be run on at least two computers running BizTalk Server (for redundancy in case one fails). For optimal performance, you should have at least one tracking host instance per MessageBox database. The actual number of tracking host instances should be (N + 1), where N = the number of MessageBox databases. The "+ 1" is for redundancy, in case one of the computers hosting tracking fails.
A tracking host instance moves tracking data for specific MessageBox databases, but there will never be more than one tracking host instance moving data for a specific MessageBox database. For example, if you have three MessageBox databases, and only two tracking host instances, then one of the host instances needs to move data for two of the MessageBox databases. Adding a third tracking host instance distributes the tracking host work to another computer running BizTalk Server. In this scenario, adding a fourth tracking host instance would not distribute any more tracking host work, but would provide an extra tracking host instance for fault tolerance.
For more information about the BAM Event Bus service, see the following topics in BizTalk Server 2006 R2 Help:
Increase the number of SOAP and HTTP adapter concurrent connections
By default, the SOAP and HTTP adapters (and .NET, in general) open only two concurrent HTTP connections from each BizTalk Server to any specific destination server. For example, if you have a SOAP send port sending messages to http://www.contoso.com/SomeWebService.asmx, then by default each BizTalk Server will open only two concurrent HTTP connections to www.contoso.com, no matter how many messages need to be sent.
This setting conforms to the IETF RFC for the HTTP 1.1 specification, and although it is suitable for user scenarios, it is not optimized for high throughput. The setting effectively throttles outbound SOAP and HTTP calls to each server to two concurrent sends from each BizTalk Server.
To increase the number of concurrent connections, you can modify the entry in the BizTalk Server configuration file, BTSNTSvc.exe.config (or BTSNTSvc64.exe.config for 64-bit hosts), on each BizTalk Server. You can increase this for the specific servers being called. For information about modifying this entry in the BTSNTSvc.exe.config or BTSNTSvc64.exe.config file, see "SOAP Adapter Configuration and Tuning Parameters" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkId=101592.
The following is an example of the configuration for the maximum connections property:
<configuration> <system.net> <connectionManagement> <add address="www.contoso.com" maxconnection="20" /> <add address="*" maxconnection="10" /> </connectionManagement> </system.net> </configuration>
|Do not increase the value for the maxconnection parameter to such a large value that the Web server being called is overwhelmed with HTTP connections. Perform stress testing by sending requests to each destination Web server to determine a value for maxconnection that will provide good performance for SOAP and HTTP sends without overwhelming the target Web servers.|
For information about tuning IIS and ASP.NET settings for Web services, see the "ASP.NET settings that can impact HTTP or SOAP Adapter performance" section of "Configuration Parameters that Affect Adapter Performance" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=79435.
Manually define worker and I/O thread values for Web applications that host orchestrations published as a Web service
When the autoConfig value in the machine.config file of an ASP.NET 2.0 application is set to true, ASP.NET 2.0 manages the number of worker threads and I/O threads that are allocated to any associated IIS worker processes:
<processModel autoConfig="true" />
The number of worker and I/O threads for an ASP.NET Web application that hosts an orchestration published as a Web service should be modified under the following conditions:
- CPU utilization is not a bottleneck on the hosting Web server.
- The value of the ASP.NET Apps v2.0.50727 \ Request Wait Time or ASP.NET Apps v2.0.50727\Request Execution Time performance counters is unusually high.
- The following error is generated in the Application log of the computer that hosts the Web application:
Event Type: Warning Event Source: W3SVC Event Category: None Event ID: 1013 Date: 6/4/2008 Time: 1:03:47 PM User: N/A Computer: <ComputerName> Description: A process serving application pool 'DefaultAppPool' exceeded time limits during shut down. The process id was '<xxxx>'.
To manually modify the number of worker and I/O threads for an ASP.NET Web application, open the associated machine.config file, and then enter new values for the maxWorkerThreads and maxIoThreads parameters:
<!-- <processModel autoConfig="true" /> --> <processModel maxWorkerThreads="200" maxIoThreads="200" />
|These values are for guidance only; ensure you test changes to these parameters.|
For more information about tuning parameters in the machine.config file for an ASP.NET 2.0 Web application, see the Microsoft Knowledge Base article 821268 “Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications” at http://support.microsoft.com/kb/821268.
Define CLR hosting thread values for BizTalk host instances
Because a Windows thread is the most basic executable unit available to a Windows process, it is important to allocate enough threads to the .NET thread pool associated with an instance of a BizTalk host to prevent thread starvation. When thread starvation occurs, there are not enough threads available to perform the requested work that negatively impacts performance. At the same time, care should be taken to prevent allocating more threads to the.NET thread pool associated with a host than is necessary. The allocation of too many threads to the .NET thread pool associated with a host may increase context switching. Context switching occurs when the Windows kernel switches from running one thread to a different thread, which incurs a performance cost. Excessive thread allocation can cause excessive context switching, which will negatively impact overall performance.
Modify the number of Windows threads available in the .NET thread pool associated with an instance of a BizTalk host by creating the appropriate CLR Hosting values in the registry of the BizTalk Server.
|Incorrect use of Registry Editor may cause problems requiring you to reinstall your operating system. Use Registry Editor at your own risk. For more information about how to back up, restore, and modify the registry, see the Microsoft Knowledge Base article "Description of the Microsoft Windows registry" at http://go.microsoft.com/fwlink/?LinkId=62729.|
|Worker threads are used to handle queued work items and I/O threads are dedicated callback threads associated with an I/O completion port to handle a completed asynchronous I/O request.|
To modify the number of threads available in the .NET thread pool associated with each instance of a BizTalk host, follow these steps:
- Stop the BizTalk host instance.
- Click Start, click Run, type regedit.exe, and then click OK to start Registry Editor.
Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$hostname] where hostname is the name of the host associated with the host instance.
Note If you have upgraded your BizTalk Server 2006 installation from BizTalk Server 2004, this registry key may be represented as HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvcguid] where guid is a GUID unique to each instance of a BizTalk Server host.
- Locate the CLR Hosting key. If this key does not exist, then create the key by following these steps:
- On the Edit menu, click New, and then click Key.
- Type CLR Hosting, and then press ENTER.
- On the Edit menu, click New, and then click Key.
- Under the CLR Hosting key, create the following DWORD entries with the indicated values.
DWORD entry Default value Recommended value
100ImportantIncreasing this value beyond 100 can have an adverse effect on the performance of the SQL Server computer hosting the BizTalk Server MessageBox database. When this problem occurs, SQL Server may encounter a deadlock condition. It is recommended this parameter is not increased beyond a value of 100.
Note These recommended values will be sufficient for most scenarios but may need to be increased depending on how many adapter handlers or orchestrations are running in each host instance. Note These values are implicitly multiplied by the number of processors on the server. For example, setting the MaxWorkerThreads entry to a value of 100 would effectively set a value of 400 on a 4 CPU server.
- Close Registry Editor.
- Restart the BizTalk host instance.
Disable tracking for orchestrations, send ports, receive ports, and pipelines when tracking is not required
Tracking incurs performance overhead within BizTalk Server as data has to be written to the MessageBox database and then asynchronously moved to the BizTalk Tracking database. If tracking is not a business requirement, then disable tracking to reduce overhead and increase performance. For more information about configuring tracking, see “Configuring Tracking Using the BizTalk Server Administration Console” in the BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106742.
Decrease the purging period for the DTA Purge and Archive job from 7 days to 2 days in high throughput scenarios
By default, the purging interval for tracking data in BizTalk Server is set to 7 days. In a high throughput scenario, this can result in an excessive build up of data in the Tracking database, which will eventually impact the performance of the MessageBox and in turn negatively impact message processing throughput.
In high throughput scenarios, reduce the hard and soft purging interval from the default of 7 days to 2 days. For more information about configuring the purging interval, see “How to Configure the DTA Purge and Archive Job” in the BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=104908.
Install the latest service packs
The latest service packs for both BizTalk Server and the .NET Framework should be installed, as these contain fixes that can correct performance issues you may encounter.
Do not cluster BizTalk hosts unless absolutely neccessary
While BizTalk Server 2006 allows you to configure a BizTalk host as a cluster resource, you should only consider doing this if you need to provide high availability to a resource that cannot be hosted across multiple BizTalk computers. As an example, ports using the FTP adapter should only reside on one host instance, as the FTP protocol does not provide file locking, however, this introduces a single point of failure which would benefit from clustering. Hosts that contain adapters, such as file, SQL, HTTP or processing only hosts, can be internally load balanced across machines and do not benefit from clustering.
By default, BizTalk Server is optimized for throughput rather than low-latency. The following optimizations can be applied to BizTalk Server in scenarios where reduced latency is required.
|These optimizations will improve latency but may do so at some cost to overall throughput.|
Increase the BizTalk Server host internal message queue size
Each BizTalk host has its own internal in-memory queue. Increase the size of this queue from the default value of 100 to 1000 to improve performance for a low-latency scenario. For more information about modifying the value of the internal message queue size, see “How to Modify the Default Host Throttling Settings” in the BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=120225.
Optimize orchestration latency by reducing the number of persistence points, if possible
The number of persistence points in an orchestration is one of the biggest factors influencing the latency of messages flowing through orchestrations. Each time the orchestration engine encounters a persistence point, it will serialize the orchestration instance into the MessageBox database and incur a latency cost. Orchestration instance serialization includes all messages and variables instantiated in the orchestration. The larger the messages and variables and the greater the number of these, the longer it will take to persist. In situations where low-latency is required, eliminate of unnecessary persistence points to significantly reduce latency. For more information about orchestration persistence points, see “Persistence and the Orchestration Engine” in the BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=108780.
Reduce the MaxReceiveInterval value in the adm_ServiceClass table of the BizTalk Server management database
BizTalk Server uses a polling mechanism to receive messages from its host queues in the MessageBox. The MaxReceiveInterval value in the adm_ServiceClass table of the BizTalk Management (BizTalkMgmtDb) database is the maximum value in milliseconds that each BizTalk host instance will wait until it polls the MessageBox. The adm_ServiceClass table contains a record for the following service types:
- XLANG/S – for BizTalk orchestration host instances
- Messaging InProcess – for in-process host instances
- MSMQT – for MSMQT adapter host instances
- Messaging Isolated – for out of process host instances, used by the HTTP, SOAP, and certain WCF receive adapter handlers
By default, this value is set to 500 milliseconds, which is optimized for throughput rather than low-latency. In certain scenarios, latency can be improved by reducing this value.
|Changes to this value impact all instances of the associated service type, therefore, take care to evaluate the impact on all host instances before changing this value.|
|This value is only used if the MessageBox has no remaining unprocessed messages. If there is a constant backlog of unprocessed messages in the MessageBox, BizTalk Server will attempt to process the messages without waiting on the polling delay. After all messages are processed, BizTalk Server will begin polling using the value specified for MaxReceiveInterval.|
|In a BizTalk Server environment with a high ratio of host instances to MessageBox database instances, decreasing the value for MaxReceiveInterval may cause excessive CPU utilization on the SQL Server computer that houses the MessageBox database instance. For example, if the MaxReceiveInterval is decreased to a low value (< 100) in a BizTalk Server environment with a single MessageBox and > 50 host instances, CPU utilization on the SQL Server may climb above 50%. This phenomenon can occur because the overhead associated with continually polling host queues is significant. If you reduce MaxReceiveInterval to a value less than 100, you should also evaluate the impact that this has on your SQL Server computer’s CPU utilization.|
Configure the MQSeries adapter using the following settings when possible to increase performance.
Adjust the value for the MQSeries receive adapter polling threads
Increase the value specified for Threads in the MQSeries Transport Properties when configuring MQSeries adapter receive locations. Increasing this property from the default value of 2 to a value of 5 will significantly improve the rate at which messages can be received using the MQSeries adapter.
Disable transaction support on MQSeries receive locations where not required
MQSeries adapter transaction support incurs significant overhead. If transaction support is not a business requirement, set the Transaction Supported value in the MQSeries Transport Properties from the default value of Yes to No.
MQSeries low-latency configuration
When using the BizTalk Adapter for MQSeries 2.0, you may notice that BizTalk Server does not maintain end-to-end latency for all messages. For more information on this problem and the steps to resolve it, see http://support.microsoft.com/kb/938986.
Apply the following recommendations from the BizTalk Server documentation as appropriate:
- “Troubleshooting MessageBox Latency Issues” at http://go.microsoft.com/fwlink/?LinkId=114747
- “Identifying Performance Bottlenecks” at http://go.microsoft.com/fwlink/?LinkID=104418
- “Avoiding DBNETLIB Exceptions” at http://go.microsoft.com/fwlink/?LinkID=108787
- “Avoiding TCP/IP Port Exhaustion” at http://go.microsoft.com/fwlink/?LinkID=101610
- “Setting the EPM Threadpool Size” at http://go.microsoft.com/fwlink/?LinkId=114748