Inbound Host Throttling

Inbound Host Throttling

Inbound host throttling, also known as message publishing throttling in BizTalk Server, is applied to the host instances that contain receive adapters or orchestrations that publish messages to the MessageBox database. An inbound host throttling condition can be triggered under the following conditions:

The amount of memory, the number of threads, or the number of database connections used by the host instance exceeds the throttling thresholds defined in the Throttling Thresholds dialog box available from the Advanced page of the Host Properties dialog box. These values are measurable with performance monitor counters available under the BizTalk:Message Agent performance object category.
Downstream hosts are unable to process the messages that are published. This increases the value of the Message count in database parameter. The threshold at which the Message count in database value triggers a throttling condition is configurable on the Throttling Thresholds dialog box available from the Advanced page of the Host Properties dialog box. The message count in the database can be measured with the Database Size counter under the BizTalk:Message Agent performance object category.
The Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate * the specified Rate overdrive factor (percent) value. The Rate overdrive factor (percent) value is defined on the Message Publishing Throttling Settings dialog box available from the Advanced page of the Host Properties dialog box. The message publishing incoming and outgoing rates can be measured with the corresponding performance monitor counters under the BizTalk:Message Agent performance object category.
The default throttling behavior has been modified by setting a registry value or values to control the throttling behavior of a host process. For more information about the registry keys for overriding the default throttling behavior, see the "Registry settings that affect default throttling behavior" topic in "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.

Depending on the severity of the throttling condition, the following actions are taken:

A progressive delay in the processing logic of the host instance is implemented. The delay may be implemented when an End Point Manager (EPM) thread receives a batch of messages from the transport adapter, and/or when the EPM submits a batch of messages to be published into the MessageBox database. Both the duration of the processing delay and the rate at which the duration is incremented scale with the severity of the throttling condition.
The number of threads that are available to the End Point Manager (EPM) is restricted. The EPM receives batches of messages from adapters and publishes the messages to the MessageBox database. By default, the EPM is configured to use 20 threads per CPU. If the host throttling mechanism detects a stress condition for inbound processing then it can temporarily reduce the number of threads available to the EPM until the stress condition is eliminated. The EPM cannot process messages from transport adapters or deliver message batches to the MessageBox database unless an EPM thread is available to service the inbound message batch.
The use of memory and other resources is reduced as applicable. BizTalk Server can send instructions to other service classes to limit memory use by dehydrating running schedules, shrinking memory cache size, and by limiting the usage of memory-intensive threads.

The following figure depicts inbound message flow from adapters to the MessageBox database.

Inbound Message flow from Adapter to MessageBox

Inbound Throttling Condition Triggers, Actions, and Mitigation Strategies

The BizTalk Server throttling mechanism applies inbound throttling to the orchestration engine (XLANG) and to inbound adapters.

Use the Message publishing throttling state and Message publishing throttling state duration counters associated with BizTalk:MessageAgent performance object category to measure the current throttling state and duration of throttling. For more information about the available host throttling performance counters, see "Host Throttling Performance Counters" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106730.

Inbound throttling can cause inbound messages to backlog at the source. If inbound throttling is applied to a receive adapter then the receive adapter may stop receiving messages until the throttling condition is mitigated. The following table describes factors involved in inbound throttling, as well as appropriate mitigation strategies.

Message Publishing Throttling State	Trigger for Throttling Condition	Throttling Actions Taken	Mitigation Strategy	Performance Object	Performance Counter
2	Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate * the specified Rate overdrive factor (percent) value. The database cannot keep up with the publishing rate.	Block the publishing thread for a dynamically computed time period until the Message Publishing Incoming Rate is at par with the Message Publishing Outgoing Rate * the specified Rate overdrive factor (percent) value.	Use performance counters to determine the message publishing incoming and message publishing outgoing rate. Consider an appropriate overdrive factor for your environment. Verify that the values supplied for the Sampling window duration (milliseconds) and Minimum number of samples parameters are appropriate for your scenario. For more information about these parameters see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent	Message publishing incoming rate Message publishing outgoing rate
4	Process memory exceeds the specified threshold. This can occur if the batch to be published has steep memory requirements or too many threads are processing messages	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. If there is a steep memory requirement to persist the messages in a batch to the database, the publishing thread is also subject to a progressive delay before the messages are persisted to the database. Whether the publishing batch will be blocked or not due to process memory pressure depends on several factors including the number of messages in the batch or if there are dehydration or message deletion commands in the batch.	Consider reducing the amount of load by reducing the EPM thread pool, and/or the size of adapter batches. If the process is not consuming excessive memory, consider increasing the Process memory usage threshold for the host. For more information about changing the Process memory usage threshold see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent	High process memory Process memory usage (MB) Process memory usage threshold (MB)
6	Host message queue size, the spool table size or the tracking table size exceed the specified threshold. Possible reasons for this condition include: The SQL Server Agent jobs used by BizTalk Server to maintain the BizTalk Server databases not running or are running slowly. Down-stream components are not processing messages from the in-memory queue in a timely manner. Number of suspended messages is high. Maximum sustainable load for the system has been reached.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Ensure that the SQL Server Agent jobs used by BizTalk Server to maintain the BizTalk Server databases are running and are not failing. Terminate and resume suspended instances as needed. Increase the default value for the Message count in database threshold taking into consideration the space requirements of the SQL server that houses the BizTalk Server databases. If your database is sized appropriately to handle additional message backlog, consider increasing the ThrottlingSpoolMultiplier and ThrottlingTrackingDataMultiplier registry values to allow additional backlog in the Spool and Tracking tables. For more information about changing the values see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent BizTalk:Message Box:General Counters BizTalk:Message Box:Host Counters	MessageAgent /Database size Message Box:General Counters /Spool size Message Box:General Counters /Tracking data size Message Box:Host Counters/Host queue – length Message Box:Host Counters/Host queue - suspended msgs – length
8	Database sessions being used by the host instance exceed the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Consider increasing the Database connections per CPU threshold for the host. For more information about changing this value, see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent	Database session
9	Process thread count exceeds the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Consider adjusting the different thread pool sizes to ensure that the system does not create a large number of threads. For more information about modifying the thread pool sizes see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent	Thread count Thread count threshold
5	System memory exceeds the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. If there is a steep memory requirement to persist the messages in a batch to the database, the publishing thread is also subject to a progressive delay before the messages are persisted to the database. Whether the publishing batch will be blocked or not due to process memory pressure depends on several factors including the number of messages in the batch or if there are dehydration or message deletion commands in the batch.	Consider reducing load by reducing the default size of the EPM thread pool, and/or the size of adapter batches. If the process is not consuming excessive memory, consider increasing the Physical memory usage threshold for the host. For more information about changing the Physical memory usage threshold, see "How to Modify the Default Host Throttling Settings" in BizTalk Server 2006 R2 Help at http://go.microsoft.com/fwlink/?LinkID=106731.	BizTalk:MessageAgent	Physical memory usage (MB) Physical memory usage threshold (MB)

Concepts

Outbound Host Throttling