How BizTalk Server Implements Host Throttling

How BizTalk Server Implements Host ...

The BizTalk Server host throttling mechanism continually monitors for a throttling condition, calculates the severity of the throttling condition, and applies host throttling progressively depending on the calculated severity. The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server processing scenarios. BizTalk Server host throttling exposes several configurable options that can be used to tune throttling for specific scenarios. For information about changing these configuration options, see How to Modify the Default Host Throttling Settings.

Components of the host throttling algorithm

BizTalk Server uses the following algorithm when applying host throttling:

Continuously monitor the following parameters to see if they exceed certain thresholds. If the values for the parameter exceed the threshold for the parameter then a throttling condition exists.
- Amount of memory in use (both systemwide and host process memory).
- Number of in-process messages being delivered or processed (threshold for outbound throttling).
- Number of threads in use.
- Database size, measured by the number of items in the queue tables for all hosts and the number of items in the spool and tracking tables.
- Number of concurrent database connections.
- Rate of message publishing (inbound) and delivery or processing (outbound).
Determine the severity of the throttling condition. Throttling conditions are ranked in severity as follows (from most severe to least severe).
- Host process memory in use exceeds threshold.
- Number of in-process messages exceeds threshold.
- Count of threads in use exceeds threshold.
- Database size exceeds threshold.
- All other throttling conditions.
Progressively apply throttling based upon the severity of the throttling condition(s). Throttling is applied more aggressively as the severity level increases. Progressive throttling is achieved as follows:
- One or more throttling conditions are detected and assigned a severity.
- An instruction to implement throttling is issued based on the condition with the highest severity. Depending on the throttling condition, if the condition persists, the size of various thread pools may be reduced and memory may be freed by dehydrating running orchestrations.
- A delay is applied in the publishing or processing of the message, depending on whether the message is inbound or outbound. The delay period is proportional to the severity of the throttling condition, so higher severity throttling conditions will initiate a longer throttling period than lower severity throttling conditions. This delay period is adjusted up and down within certain ranges by the throttling mechanism as conditions change. The current delay period is exposed through the Message delivery delay (ms) and the Message publishing delay (ms) performance counters associated with the BizTalk:Message Agent performance object category. These performance object counters are documented in the topic Host Throttling Performance Counters.
- The throttling mechanism continues to check if a throttling condition is present. If the throttling conditions are mitigated then the throttled messages are unblocked and the thread pool and other resources are allowed to operate in an unrestricted mode. If the system continues to operate without any throttling conditions, the delay period is greatly reduced. If the throttling condition persists, the delay amount is incremented proportional to the severity of the condition and subsequent messages are subject to the higher delay.
- Throttling is no longer implemented when the delay period has elapsed.

Type of throttling conditions

There are three primary types of throttling conditions: rate based, resource based, and user controlled.

Rate based throttling is divided into two categories; inbound (published) and outbound (delivered):
- For inbound (published) messages, BizTalk Server throttles publishing of messages if The Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate * the specified Rate overdrive factor (percent) value. The Rate overdrive factor (percent) parameter is configurable on the Message Publishing Throttling Settings dialog box. Rate based throttling for inbound messages is accomplished primarily by inducing a delay before publishing the batch of messages into the MessageBox database. No other action is taken to accomplish rate based throttling for inbound messages.
- For outbound (delivered) messages, BizTalk Server throttles delivery of messages if The Message delivery incoming rate for the host instance exceeds the Message delivery outgoing rate * the specified Rate overdrive factor (percent) value. The Rate overdrive factor (percent) parameter is configurable on the Message Processing Throttling Settings dialog box. Rate based throttling for outbound messages is accomplished primarily by inducing a delay before removing the messages from the in-memory queue and delivering the messages to the End Point Manager (EPM) or orchestration engine for processing. No other action is taken to accomplish rate based throttling for outbound messages.
  
  For more information about the rate overdrive factor see How to Modify the Default Host Throttling Settings.
Resource based throttling monitors system resources such as threads, memory, and database size and can be applied to any service class.
User controlled throttling occurs when the default throttling behavior has been modified by setting a registry value or values to control the throttling behavior of a host process. The registry keys for overriding the default throttling behavior are described in the "Registry settings that affect default throttling behavior" section of the topic How to Modify the Default Host Throttling Settings.

Throttling condition triggers, actions, and mitigation strategies

This section describes the triggers for various throttling conditions, the resultant actions by the throttling mechanism, and techniques that may be employed to mitigate the throttling condition.

Inbound Throttling

The BizTalk throttling mechanism applies inbound throttling to the orchestration engine (XLANG) and to inbound adapters.

Use the Message publishing throttling state and Message publishing throttling state duration counters associated with BizTalk:MessageAgent performance object category to measure the current throttling state and duration of throttling. For more information about the available host throttling performance counters, see Host Throttling Performance Counters.

Inbound throttling can cause inbound messages to backlog at the source. If inbound throttling is applied to a receive adapter then the receive adapter may stop receiving messages until the throttling condition is mitigated.

Message Publishing throttling state	Trigger for throttling condition	Throttling actions taken	Mitigation strategy	Performance object	Performance counter
2	Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate * the specified Rate overdrive factor (percent) value. The database cannot keep up with the publishing rate.	Block the publishing thread for a dynamically computed time period until the Message Publishing Incoming Rate is at par with the Message Publishing Outgoing Rate * the specified Rate overdrive factor (percent) value.	Use performance counters to determine the message publishing incoming and message publishing outgoing rate. Consider an appropriate overdrive factor for your environment. Verify that the values supplied for the Sampling window duration (milliseconds) and Minimum number of samples parameters are appropriate for your scenario. For more information about these parameters see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Message publishing incoming rate Message publishing outgoing rate
4	Process memory exceeds the specified threshold. This can occur if the batch to be published has steep memory requirements or too many threads are processing messages	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. If there is a steep memory requirement to persist the messages in a batch to the database, the publishing thread is also subject to a progressive delay before the messages are persisted to the database. Whether the publishing batch will be blocked or not due to process memory pressure depends on several factors including the number of messages in the batch or if there are dehydration or message deletion commands in the batch.	Consider reducing the amount of load by reducing the EPM thread pool, and/or the size of adapter batches. If the process is not consuming excessive memory, consider increasing the Process memory usage threshold for the host. For more information about changing the Process memory usage threshold see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	High process memory Process memory usage (MB) Process memory usage threshold (MB)
6	Host message queue size, the spool table size or the tracking table size exceed the specified threshold. Possible reasons for this condition include: The SQL Agent jobs used by BizTalk Server to maintain the BizTalk Server databases not running or are running slowly. Down-stream components are not processing messages from the in-memory queue in a timely manner. Number of suspended messages is high. Maximum sustainable load for the system has been reached.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Ensure that the SQL Agent jobs used by BizTalk Server to maintain the BizTalk Server databases are running and are not failing. Terminate and resume suspended instances as needed. Increase the default value for the Message count in database threshold taking into consideration the space requirements of the SQL server that houses the BizTalk databases. If your database is sized appropriately to handle additional message backlog, consider increasing the ThrottlingSpoolMultiplier and ThrottlingTrackingDataMultiplier registry values to allow additional backlog in the Spool and Tracking tables. For more information about changing the values see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent BizTalk:Message Box:General Counters BizTalk:Message Box:Host Counters	MessageAgent /Database size Message Box:General Counters /Spool size Message Box:General Counters /Tracking data size Message Box:Host Counters/Host queue – length Message Box:Host Counters/Host queue - suspended msgs – length
8	Database sessions being used by the host instance exceed the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Consider increasing the Database connections per CPU threshold for the host. For more information on changing this value see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Database session
9	Process thread count exceeds the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. The publishing thread is also subject to a progressive delay before the messages are persisted to the database.	Consider adjusting the different thread pool sizes to ensure that the system does not create a large number of threads. For more information about modifying the thread pool sizes see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Thread count Thread count threshold
5	System memory exceeds the specified threshold.	Reduce the size of the thread pool used by EPM. Block the EPM threads to prevent processing of new messages batches. If there is a steep memory requirement to persist the messages in a batch to the database, the publishing thread is also subject to a progressive delay before the messages are persisted to the database. Whether the publishing batch will be blocked or not due to process memory pressure depends on several factors including the number of messages in the batch or if there are dehydration or message deletion commands in the batch.	Consider reducing load by reducing the default size of the EPM thread pool, and/or the size of adapter batches. If the process is not consuming excessive memory, consider increasing the Physical memory usage threshold for the host. For more information on changing the Physical memory usage threshold see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Physical memory usage (MB) Physical memory usage threshold (MB)

Outbound Throttling

The BizTalk throttling mechanism applies outbound throttling to the orchestration engine (XLANG) and to outbound adapters.

Use the Message delivery throttling state and Message delivery throttling state duration counters associated with BizTalk:MessageAgent performance object category to measure the current throttling state and duration of throttling. For more information about the available host throttling performance counters see Host Throttling Performance Counters.

Outbound throttling can cause delayed message delivery and messages may build up in the in-memory queue and cause de-queue threads to be blocked until the throttling condition is mitigated. When de-queue threads are blocked no additional messages are pulled from the MessageBox into the in-memory queue for outbound delivery.

Message Delivery throttling state	Trigger for throttling condition	Throttling actions taken	Mitigation Strategy	Performance Object	Performance Counter
1	Message delivery incoming rate for the host instance exceeds the Message delivery outgoing rate * the specified Rate overdrive factor (percent) value This can be caused by high processing complexity, slow outbound adapters, or a momentary shortage of system resources.	Block the delivery thread for a dynamically computed time period until the Message delivery incoming rate is at par with the Message delivery outgoing rate * the specified Rate overdrive factor (percent) value.	Use performance counters to determine the message delivery incoming and message delivery outgoing rate. Consider an appropriate over drive factor for your environment. Verify that the values supplied for the Sampling window duration (milliseconds) and Minimum number of samples parameters are appropriate for your scenario. For more information about these parameters see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Message delivery incoming rate Message publishing outgoing rate
4	Process memory exceeds the specified threshold. This can occur in memory intensive processing scenarios, when processing large messages, or when send adapters are attempting to process a high number of messages simultaneously.	Slow down message delivery to adapters or XLANG. Reduce process memory consumption by dehydrating service instances and shrinking cache when applicable. Reduce the size of the thread pools used by the EPM and/or the Message Agent. Periodically force a .NET garbage collection (GC).	If the system is not becoming idle due to process memory based throttling, no action may be required. If the In-process message count value is high and the CPU utilization is not excessive even when there is process memory based throttling, no additional action may be required. If the system appears to be over-throttling, consider increasing the value associated with the Process memory usage threshold for the host and verify that the host instance does not generate an "out of memory" error. If an "out of memory" error is raised by increasing the Process memory usage threshold, then consider reducing the values for the Internal message queue size and In-process messages per CPU thresholds. This strategy is particularly relevant in large message processing scenarios.	BizTalk:MessageAgent	High process memory Process memory usage(MB) Process memory usage threshold (MB) In-process message count Active instance count
3	Number of in-process messages delivered to a service class exceeds the specified threshold. This can be caused by high processing complexity, slow outbound adapters, or a momentary shortage of system resources.	Slow down message delivery to adapters or XLANG. Reduce the size of the thread pool used by the Message Agent.	If over-throttling occurs, consider increasing the value associated with the In-process messages per CPU threshold. Note Increasing this value may adversely impact send adapter performance and/or increase the memory usage of the process.	BizTalk:MessageAgent	In-process message count In-process message count threshold
9	Process thread count exceeds the specified threshold.	Reduce the size of the thread pools used by the EPM and/or the Message Agent	Consider adjusting the different thread pool sizes to ensure that the system does not create a large number of threads. For more information about modifying the thread pool sizes see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Thread count Thread count threshold
5	System memory exceeds a threshold.	Slow down message delivery to adapters or XLANG. Reduce process memory consumption by dehydrating service instances and shrinking cache when applicable. Reduce the size of the thread pools used by the EPM and/or the Message Agent.	Consider reducing load by reducing the default size of the EPM thread pool, and/or the size of adapter batches. If the process is not consuming excessive memory, consider increasing the Physical memory usage threshold for the host. For more information about changing the Physical memory usage threshold see How to Modify the Default Host Throttling Settings.	BizTalk:MessageAgent	Physical memory usage (MB)