To assess the scalability of the MessageBox database, the testing involved one SQL Server computer (hosting the MessageBox database) and multiple BizTalk Server computers. Different tests were run with varying numbers of CPU processors (one, two, four, and eight CPUs) on the single SQL Server. The following figure shows the deployment topology for the scalability testing of the MessageBox database in BizTalk Server 2004.

Figure 5 Deployment topology for MessageBox database scalability testing

Hardware Configuration

The objective for this series of tests was to measure the processing speed and throughput of the MessageBox database as CPU processors were added to the SQL Server computer. To show the scalability of the message box with varying numbers of CPU processors on the SQL Server, a single messaging scenario was used in all of the tests (the only variable was the number of CPU processors on the SQL Server).

Because the message box is the engine for message publishing and routing, the messaging scenario used for the performance testing is sufficient for generating workload. Orchestration is unnecessary because it has less-significant performance impact on the message box.

The scenario involves a file receive adapter that receives a purchase order request and routes the order to various departments based on the content of the request. Various files ranging from 2 to 6 KB in size are dropped to the file receive location at periodic intervals. To isolate and measure the performance of the message box on the SQL Server computer, the messaging workload on the BizTalk Server computers was maximized for the duration of the test runs. The following figure shows the data flow diagram for the scalability testing of the message box in BizTalk Server 2004.

Figure 6 Data flow diagram for message box scalability testing

Scaling up the SQL Server from one to two CPU processors improved end-to-end throughput (documents per second) by 90 percent. End-to-end throughput is the total time from message reception (when a file is dropped in the receive directory) to message transmission (when a file is dropped in the outbound directory). Scaling up from two to four CPU processors improved performance by approximately 85 percent for receiving and 50 percent for processing. When scaling up from two to four CPU processors, two more BizTalk Server computers were added to the deployment configuration to generate additional workload on the four-processor SQL Server that hosts the MessageBox database.

Scaling up the SQL Server from four to eight CPU processors provided a 20 to 30 percent increase in performance. As more BizTalk Server computers are added to the deployment configuration, the message box eventually becomes the bottleneck and SQL Server lock contentions increase. At this point, adding BizTalk Server computers to the deployment does not improve throughput. The following table shows the performance data from the scalability testing of the message box in BizTalk Server 2004.

Table 3 Performance data for message box scalability testing

This section explains how overall system throughput is affected by various message box configurations. It also includes some significant findings from the performance testing (such as the optimal number of BizTalk Servers per message box, and why scaling the message box from one to two SQL Server computers might not necessarily improve performance). The following table shows the performance impact on throughput with different numbers of message boxes.

Table 4 Performance impact on throughput

Single Message Box

Multiple Message Boxes

The scenario for this series of tests is the same scenario as the one used for the scalability testing of BizTalk Server 2004 in a messaging scenario (described in the earlier topic, "Scaling Out BizTalk Server in a Messaging Scenario"), except that this scenario involves three message boxes and includes dedicated servers for processing orchestrations. For inbound traffic, the File adapter is used with an XML disassembler and XML assembler pipeline component. For port configuration, one receive port (one receive location) and one send port are configured. The orchestration is a simple schedule involving decision and transformation. Other details about this scenario are the same as in the messaging scenario.

The following figure shows the data flow diagram for the scalability testing of BizTalk Server 2004 in an orchestration scenario.

Figure 9 Data flow diagram for BizTalk Server 2004 scalability testing

This series of tests used both paths in the decision tree: one half of received documents were accepted, and the other half were rejected.

The following table compares the end-to-end throughput statistics between the orchestration and messaging scenarios. End-to-end throughput is the total time from message reception (when a file is dropped in the receive directory) to message transmission (when a file is dropped in the outbound directory).

Table 6 Comparison of end-to-end throughput statistics

With one BizTalk Server computer, all BizTalk Server functionality (receiving, sending, and processing) runs on the same server. Scaling out from one to two BizTalk Server computers provides near-linear scaling because orchestration processing is isolated onto a dedicated BizTalk Server computer.

For this scenario, scaling out from two to three BizTalk Server computers improves throughput minimally because in this configuration, the receiving and sending functionality are isolated onto separate dedicated servers. Therefore, there is still only one dedicated server for processing orchestrations. If CPU processor utilization and memory consumption are not maximized on the receive and send servers, then it is unnecessary to add a dedicated server for processing orchestrations because you can run orchestration on the receive and send servers.

Scaling out from three to six BizTalk Server computers provides near-linear scaling because each of the three primary areas of functionality (receive, send, and orchestration) has twice the processing resources. Throughput reaches a peak when scaling out from six to seven BizTalk Server computers because beyond seven BizTalk Server computers, the three orchestration servers cause bottlenecks on the MessageBox databases. For this scenario, the end-to-end throughput rate decreases from 256.68 documents per second with seven BizTalk Server computers to 241.21 documents per second with eight BizTalk Server computers.

Scaling out from seven to eight BizTalk Server computers causes slight performance degradation in overall throughput because the eighth BizTalk Server computer in the deployment configuration is the fourth orchestration server, which only adds contention on the message box. With more orchestration servers, the message box becomes the bottleneck with an increase on the following SQL Server performance counters:

• SQL Server Lock Wait Time. Ideally, less than 100 milliseconds
  
• SQL Server Lock Time Out. Ideally, less than 100 milliseconds
  
• Current Disk Queue Length. Ideally, the current disk queue length minus the number of spindles should be less than 2 (that is, fewer than two queued items per spindle is good)
  
• Percent CPU utilization. Ideally, 70 to 75 percent
  
• Processor Queue Length. Ideally, 10 per processor
  

To improve the throughput and scalability of BizTalk Server 2004 in an orchestration scenario involving more than three orchestration servers, scale up the database tier by upgrading to faster CPU processors, adding CPU processors, and including additional memory.

The objective for this series of tests was to measure the processing speed and throughput of BizTalk Server 2004 as more computers are added to the deployment. To show the scalability of BizTalk Server 2004 with different numbers of BizTalk Server and SQL Server computers, a single messaging scenario was used in all of the tests. The only variables were the numbers of servers.

The messaging scenario involves using the FileGen.exe tool to drop 12,000 documents (ranging from 2 to 6 KB in size) to each BizTalk Server (receive server) at periodic intervals. For every document that BizTalk Server receives, two documents are processed and sent. To maintain optimal throughput, the messaging workload on the BizTalk Server computers was maximized for the duration of the test runs. Adding excessive workload to the receive servers causes excessive lock contentions on the MessageBox database.

The BizTalk Server receiving, processing, transmitting, and tracking features were isolated onto separate hosts. The testing also involved the following configurations:

• All databases were created on a dedicated SQL Server computer with high-speed storage area network (SAN) drives.
  
• All database and log files were stored on separate disks.
  
• Each message box was pre-sized to 10.0 GB for database files and 5.0 GB for log files.
  
• For multiple SQL Server computers hosting MessageBox databases, publishing was disabled on the primary message box. Other message boxes handled all publishing so that the primary message box was dedicated to message routing.
  
• The SQL Server Agent service was running on all SQL Server computers.
  
• The MessageBox and Tracking databases were deleted after each test run.
  
• All BizTalk Server service instances were restarted after each test run.
  

The messaging scenario does not use orchestration. For inbound traffic, the File adapter was used with an XML disassembler and a pass-through pipeline component.

For port configuration, one receive port (one receive location) and four send ports (with filters) were configured. Three maps on the send ports transformed the data based on the filter (which checked for the <MsgInd> promoted property).

The data flow diagram used for the scalability testing of BizTalk Server 2004 in a messaging scenario is the same as the data flow diagram used in the scalability testing of the message box, which is described in the earlier topic, "Scaling Up SQL Server Scenario Description."

The following table shows the throughput, in documents per second, of various deployments involving different numbers of BizTalk Server and SQL Server computers (without tracking).

Table 7 Throughput of various deployments

For the tests involving multiple BizTalk Server computers, the BizTalk Server configuration used an equal number of receive and send computers. For example, in the tests with two BizTalk Server computers, one computer was dedicated as the receive server while the other was dedicated as the send server. With four BizTalk Server computers, there were two receive servers and two send servers. The same configuration was used for the tests involving six, eight, and ten BizTalk Server computers.

With one SQL Server computer hosting the message box, throughput scales linearly up to four BizTalk Server computers. When more than four BizTalk Server computers run message box functionality against the same SQL Server computer, the SQL Server computer becomes the bottleneck due to SQL Server lock contentions on the MessageBox database. Because the message box on SQL Server handles all message publishing and routing for BizTalk Server, it encounters performance degradation from the overhead of running message box functionality for multiple BizTalk Server computers.

To scale beyond four BizTalk Server computers, a deployment must have more than one SQL Server computer for the message box. Scaling out the database tier by including additional SQL Server computers for running the message box helps maintain linear scaling for BizTalk Server 2004.

With two SQL Server computers running the message box, throughput scales linearly up to six BizTalk Server computers. When more than six BizTalk Server computers run message box functionality against the two SQL Server computers, the SQL Server computers become the bottleneck due to SQL Server lock contentions on the MessageBox databases. To scale beyond six BizTalk Server computers in a deployment with two SQL Server computers, include additional SQL Server computers for running the message box.

With three SQL Server computers running the message box, throughput scales linearly up to eight BizTalk Server computers. When more than eight BizTalk Server computers run message box functionality against the three SQL Server computers, the SQL Server computers become the bottleneck due to SQL Server lock contentions on the MessageBox databases. To scale beyond eight BizTalk Server computers in a deployment with three SQL Server computers, include additional SQL Server computers for running the message box.

With four SQL Server computers running the message box, throughput scales linearly up to 10 BizTalk Server computers. When more than 10 BizTalk Server computers run message box functionality against the four SQL Server computers, the SQL Server computers become the bottleneck due to SQL Server lock contentions on the MessageBox databases. To scale beyond 10 BizTalk Server computers in a deployment with four SQL Server computers, include additional SQL Server computers for running the message box.

This test establishes a performance baseline for a basic orchestration in BizTalk Server 2004. The basic orchestration tested contains one receive shape and one send shape (File transport). This orchestration provides the highest performance because it does no processing; it only receives and sends messages. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 11 Basic orchestration diagram

This test measures the performance overhead of adding a simple filter expression to an orchestration (where the receive shape has the Activate property set to True). This orchestration contains one receive shape and one send shape (File transport). The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 12 Simple filter expression orchestration diagram

This test measures the performance overhead of using a blank scope in an orchestration to perform some simple in-memory operations. The orchestration contains one receive shape and one send shape (File transport).

Compared to the previous orchestrations tested, orchestrations with an atomic scope have higher CPU utilization because they require additional communication with the MessageBox database for data persistence. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 13 Orchestration with an atomic scope

While the previous orchestration uses a blank scope, this orchestration contains an expression shape that instantiates a COM+ object, which executes a query against a remote database. The File transport is used for both receive and send ports. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 14 Orchestration with an atomic scope using DTC

A sequential convoy has two receive shapes bound to the same port. The orchestration uses a correlation with one promoted property. The File transport is used for both receive and send ports. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 15 Orchestration with a sequential convoy

To test performance, the inbound publishing rate is increased until CPU utilization on the SQL Server (hosting the MessageBox database) reaches 100 percent.

Dehydration is a state that results from high memory consumption, and leads BizTalk Server 2004 to unload orchestrations from memory and persist them to disk. While other types of orchestrations (such as an orchestration with an atomic scope or an orchestration with a sequential convoy) can have dehydration, they might not dehydrate if sufficient memory is available. This scenario causes memory contention to impose dehydration.

This orchestration contains a one-minute delay shape that causes it to dehydrate after several other instances are created. The File transport is used for both receive and send ports. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 16 Orchestration with dehydration

The performance is volatile for orchestrations with dehydration because dehydration and rehydration occur frequently in this scenario. End-to-end throughput reached an average of 37.2 orchestrations dehydrated/second, and 39 orchestrations rehydrated/second.

For this test, both orchestrations run under the same host. The File transport is used for both receive and send ports. The following figure shows the orchestration diagrams in BizTalk Server 2004 Orchestration Designer.

Figure 17 Orchestration calling another orchestration

For this test, both orchestrations run under the same host. The File transport is used for both receive and send ports. The following figure shows the orchestration diagrams in BizTalk Server 2004 Orchestration Designer.

Figure 18 Orchestration starting another orchestration

For this test, the orchestration uses one promoted property and one distinguished field. Five hundred different documents are sent, resulting in 500 unique orchestration instances (with batches of five messages being sent). The File transport is used for both receive and send ports. The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 19 Singleton orchestration with batched messaging

To test performance, the inbound publishing rate is increased until CPU utilization on the SQL Server (hosting the MessageBox database) reaches 100 percent. The performance is primarily limited by the CPU on the SQL Server (that hosts the MessageBox database) because orchestrations with sequential convoys use a publish/subscribe model (where latency is inherent).

For this test, the orchestration uses a two-way port (request-response) with a Web services adapter (SOAP). The following figure shows the orchestration diagram in BizTalk Server 2004 Orchestration Designer.

Figure 20 Orchestration exposed as a Web service

To test performance, the inbound publishing rate is increased until CPU utilization on the SQL Server (hosting the MessageBox database) reaches 100 percent.

To measure the throughput and resource utilization from file reception (receive) to outbound transmission (send), files of various sizes were sent from a client computer. The data flow diagram used for the performance testing of the File adapter in BizTalk Server 2004 is the same as the data flow diagram used for the scalability testing of the message box, which is described in the earlier topic, "Scaling Up SQL Server Scenario Description." To set an upper limit and determine the maximum file size that the File adapter can support, a series of tests were run to measure the latency of receiving and sending various files ranging from 1 GB to 4 GB in size.

Ultimately, the maximum file size limitation is a restriction imposed by the physical resources on the system. Scaling up the hardware (such as faster CPU processors, increased memory, and larger hard disks) can increase the maximum file size supported. The File adapter processed a 1.0 GB file in 21 minutes, and a 2.0 GB file in 40.3 minutes. The following table shows the throughput for the various file sizes tested.

Table 9 Throughput for tested file sizes

When file sizes are small, throughput is limited by CPU processing and the rate at which the system can perform basic file operations. As file sizes grow, throughput (KB/sec) increases but overall performance (files/sec) decreases.

Toward the upper limit of the maximum file size limitation (typically around 5.0 GB), the receiving host becomes the bottleneck because it must parse incoming files, compress the data, and persist the data into the MessageBox database. In situations where system throughput is limited by the receiving host, the CPU utilization on other hosts (orchestration and send) begins to decrease. The following table shows the resource utilization on the BizTalk Server and SQL Server for various file sizes.

Table 10 Resource Utilization for various file sizes

The following table shows the throughput of the File adapter with and without orchestration.

Table 11 Throughput of the File adapter with and without orchestration

To optimize performance for the outbound transport, the send host is separated from the orchestration host. The performance of the HTTP adapter is similar to the performance of the File adapter up to 200 MB (throughput increases with file size). Beyond 200 MB, throughput decreases rapidly because large HTTP POST requests are divided into fragments before being sent. At the destination, the fragments are combined to build the message. The following table shows the throughput for the HTTP adapter.

Table 12 Throughput for the HTTP adapter

As file size increases, so does the time required to upload the file. Therefore, you should carefully adjust the connection time-out settings in Internet Information Services (IIS) when sending large files through the HTTP adapter. For information about modifying the connection settings, see the IIS documentation.

The following table describes the registry settings that affect the performance of the HTTP adapter. By default, the registry has no HTTP adapter keys.

Table 13 Registry settings that affect HTTP adapter performance

Create these DWORD registry keys in the following registry locations:

• For the HttpBatchSize and RequestQueueSize keys,
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc.3.0\HttpReceive
  
• For the other keys,
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{GUID} where GUID is the ID of the host for the HTTP send handler
  

The scenario for the performance testing of the Web services adapter is the same as the one used for testing the File and HTTP adapters, except that the orchestration is exposed as a Web service. The inbound adapter is exposed through SOAP while outbound traffic uses the File adapter. Configuring the scenario this way helps to isolate and measure the throughput with SOAP as the inbound adapter.

The performance of the Web services adapter is similar to the performance of the File adapter (throughput increases with file size). With large file sizes (larger than 200 MB), throughput decreases rapidly because the default settings in IIS cause retries and time-out errors. Performance tests for the Web services adapter were run with file sizes up to 200 MB. The following table shows the throughput for the Web services adapter.

Table 14 Throughput for the Web services adapter

The schema is simplified for this scenario because the receive adapter cannot support multidimensional arrays (deeply-nested schemas). By default, the maximum file size that can be uploaded by IIS is 4 MB to restrict possible denial-of-service attacks. For information about increasing and modifying this setting, go to http://go.microsoft.com/fwlink/?linkid=27097.

For this scenario, inbound traffic flows through the File adapter while outbound traffic flows through the BizTalk Message Queuing (also known as MSMQT) adapter. Because it is difficult to scale a single queue, a total of eight inbound queues and four outbound queues were used to demonstrate the full capacity of the BizTalk Message Queuing adapter. Sending messages to multiple queues requires modification to the application design. The following figure shows the data flow diagram for the performance testing of the BizTalk Message Queuing adapter in BizTalk Server 2004.

Figure 22 Data flow diagram for BizTalk Message Queuing adapter

As with the File adapter, the throughput of the BizTalk Message Queuing adapter increases with file size. As file sizes increase, the receiving host becomes the bottleneck. While the testing used file sizes up to 1 GB, there is no maximum file size limitation. File size is primarily limited by the amount of physical memory on the system as the Message Queuing (also known as MSMQ) client loads the message into memory. The following table shows the resource utilization on the BizTalk Server and SQL Server for various file sizes.

Table 15 Resource utilization for various file sizes

The following table shows the throughput for the BizTalk Message Queuing adapter.

Table 16 Throughput for the BizTalk Message Queuing adapter

To have multiple BizTalk Servers process BizTalk Message Queuing requests, configure multiple receive locations (multiple BizTalk Message Queuing queues) so that each thread can process requests in parallel (concurrently).

This section provides an overview of the scenario for testing the SQL adapter, and explains the test results for both inbound and outbound adapters. For this series of tests, the SQL adapter retrieves data from a dedicated input/output server, and not the message box. Instead of using the SQL adapter for both inbound and outbound traffic, the tests isolate the adapter for each scenario for measurement accuracy and reliability.

Inbound Adapter

Outbound Adapter

For this scenario, the inbound batch size was set to 20 through the Batch: Maximum Files property. The File adapter performs approximately 10 times faster than the FTP adapter because the FTP adapter reads messages sequentially, instead of concurrently, for each receive location. By default, the incoming batch size is set to 0, leading the adapter to enumerate and retrieve all of the files at once. When there are large numbers of files in the FTP site, the default configuration results in poor performance. Plan for your volume of traffic and set the batch size appropriately; the recommendation is 20. The following table shows the throughput for the FTP adapter.

Table 21 Throughput for the FTP adapter

The FTP adapter supports multiple locations in multiple threads, but reads files from a single location sequentially.

The most common scenario for SMTP involves using the SMTP adapter for outbound traffic. For this scenario, the File adapter is used for inbound traffic and the SMTP adapter is used for outbound traffic.

For outbound traffic, the SMTP adapter consumes more memory than the File adapter, with the SMTP adapter providing 50 percent less throughput than the File adapter. There is no file-size limitation for the SMTP adapter because file size is limited by the physical resources on the system (especially memory), and the number of concurrent SMTP messages loaded simultaneously.

As file sizes increase (greater than 2 MB), the system might encounter out-of-memory errors under high load. Therefore, adjust the watermark threshold (decrease the high-low watermark threshold) for the messaging agent when using the SMTP adapter for outbound traffic. The following table shows the resource utilization on the BizTalk Server and SQL Server for various file sizes.

Table 22 Resource utilization for various file sizes

The following table shows the throughput for the SMTP adapter.

Table 23 Throughput for the SMTP adapter

This section provides performance information about the throughput associated with different types of transport adapters in a typical messaging scenario. The testing involves one BizTalk Server and one SQL Server. The deployment topology used for testing the transport adapters in a messaging scenario is the same as the topology used for testing the transport adapters, which is described in the earlier topic, "Transport Adapters."

Scenario Description

The four transport adapter configurations tested had approximately the same performance, with outbound map transformation causing the bottleneck. The following chart shows the throughput for the different transport adapters tested in the messaging scenario.

Figure 23 Throughput for transport adapters

For the BizTalk Message Queuing adapter, the testing involved three incoming queues (BizTalk Server receive locations) to optimize thread utilization.

This section shows a sample case study based on the most basic real-world scenarios. It provides performance information about Secure Sockets Layer (SSL) overhead in a typical messaging scenario.

The testing consisted of the following computers:

• One BizTalk Server for receive
  
• One BizTalk Server for orchestration
  
• One BizTalk Server for send
  
• One SQL Server for all databases
  
• One client computer for generating requests
  
• One client computer for receiving responses
  

The following figure shows the deployment topology for the testing of SSL throughput in a messaging scenario.

Figure 24 Deployment topology for SSL testing

Scenario Description

To measure the performance impact of using SSL, tests were run with the following transport adapter configurations:

• HTTP to HTTP
  
• SOAP to File
  

A basic orchestration was used for testing the HTTP adapter. The following figure shows the data flow diagram for the HTTP scenario.

Figure 25 Data flow diagram for HTTP

For the inbound port in the SOAP-to-File scenario, the same orchestration used for testing the HTTP adapter was exposed through SOAP as a Web service. For outbound traffic, the File adapter drops outbound messages to a designated folder. The following figure shows the data flow diagram for the SOAP-to-File scenario.

Figure 26 Data flow diagram for SOAP-to-File

Test Results

This scenario involves a simple messaging implementation with no orchestrations. Separate hosts are created for send, receive, and tracking functions.

Each component was tested for its individual effect on performance. To measure this difference, a baseline performance was determined by using the default pass-through pipelines; no orchestrations or maps were included. Two sets of tests were conducted. The following tables show the permutations tested for both the receive/inbound side and the send/outbound side.

This series of tests was run against a simple orchestration with a receive shape and a send shape using the File transport adapter. This scenario includes the following BizTalk Server 2004 host configurations:

• One BizTalk Server for receive
  
• One BizTalk Server for send
  
• One BizTalk Server for orchestrations
  

The data flow diagram used for the performance testing of pipelines in BizTalk Server 2004 was the same as the data flow diagram used for the scalability testing of the message box, which is described in the earlier topic, "Scaling Up SQL Server Scenario Description."

The purpose of the tests in this section is to demonstrate the performance impact due to different levels of complexity of the schema. To maximize the workload on the XML Assembler/Disassembler, files of relatively small sizes were used. Large-sized files would not stress the XML Assembler/Disassembler efficiently because it uses more resources for compressing and persisting the data to the message box than for assembling and disassembling the XML data. Therefore, it would be difficult to measure and observe how schema complexity impacts performance.

Both the breadth and depth of XML fields in a schema have an impact on performance. As the number of fields in the XML file grows, the throughput decreases. Similarly, as the depth of the schema increases, the throughput also decreases.

For the following tests, the inputs were adjusted to maximize the CPU utilization of the BizTalk Server (to get the maximum throughput). To measure the impact that the breadth and depth of XML schemas have on performance, two test permutations were used. The following paragraph lists the different inputs used for both permutations.

XML schema complexity in breadth

• Each document for this particular test is 20 KB. The variable is the number of the fields for each document. For this test, the following inputs were used:
  
• 1 field
  
• 2 fields
  
• 100 fields
  
• 1000 fields
  

XML schema complexity in depth

• 2 levels
  
• 20 levels
  
• 200 levels
  
• 2000 levels
  

The following table shows the performance of BizTalk Server with various XML file complexities.

Table 29 Performance of BizTalk Server with XML file complexities

Note
Schemas that contain up to one million fields and are up to 199 MB in size have been tested; no maximum limit for the breadth was found.

The following table shows the relationship between the schema complexity in breadth and overall throughput.

Table 30 Relationship of breadth and overall throughput

The scenario is a simple messaging scenario that has an outbound pipeline connected directly to an inbound pipeline. The hosts for the two parts of the scenario are separated so that the outbound pipeline can be stopped without affecting the inbound pipeline. The HTTP protocol is used for inbound message traffic because it allows more control of the workload.

To focus on the performance impact based on the file size, a simple schema was used for this testing. The test files were built by repeating records to fit a size pattern. The test runs consisted of sending the test messages to the inbound port and measuring the publishing time.

The following table shows how the throughput and processing capacity in KB/second change with the file size.

Table 31 File size changes in throughput and processing capacity

The amount of data goes flat at a certain point because the time spent in communication and database access is almost insignificant compared to the actual processing time of the messages

Schema Complexity

The base schema used for the tests is a schema with 40 nodes. The maximum depth of the schema is six nodes and all the nodes are required; some nodes will repeat. All the schemas used are derived from this base. The constraints used to create the four schemas are whether all fields are optional or required, and whether the parser optimization is set for speed or complexity. The message used for measurement is 500 KB in size.

The parsing mode is an attribute on the schema Info record, with two modes: speed and complexity. In speed mode, the parser tries to fit data as it appears in the stream. In complexity mode, the flat-file parsing engine uses both top-down and bottom-up parsing, and tries to fit data more accurately. The following table shows the results of the test.

Table 32 Schema complexity test results

Schema Type	Parser Optimization	Processing Rate per Minute	Cost/Message (megacycles)
No optional fields	Speed	44.14	8369
No optional fields	Complexity	43.15	8559
All fields optional	Speed	40.42	9227
All fields optional	Complexity	26.21	14665

The cost is given in processor megacycles per second and is calculated as:

This formula gives a machine-independent cost. The variables are defined as follows:

• F = CPU frequency in MHz (megacycles/second executed by a CPU)
  
• N = number of physical CPUs on the computers
  
• p = CPU utilization (percent)
  
• k = 1.3 coefficient recommended by Intel when hyperthreading is enabled
  
• r = throughput rate in number of units (messages) processed per second.
  

From the tests, we observed that the performance decreases as the complexity of the schema grows. This result occurs because the parser retries parsing to find the best match when the schema has ambiguous definitions.

In case of an ambiguous schema, the complex parser setting will have impact on the general throughput because of the best match algorithm used during complex parsing. In the given runs there is almost a 50 percent drop in throughput for the given input file just by setting the optimization to complex.

• The same complex data definitions may cause an impact on loading time and memory consumption. At load time the parser grammar and parse table are generated automatically and the number of states the parser requires increases dramatically. That is why, for certain large schemas (a few megabytes in size), you can expect schema loading time to grow from minutes to hours.
  

Use the following procedure to set the BatchSize value. This value is the desired maximum number of messages per thread that should be received before BizTalk Server processes the interchange.

To set the BatchSize value

1. Click Start, point to All Programs, point to Microsoft Visual Studio .NET 2003, and then click Microsoft Visual Studio .NET 2003.
   
2. In Visual Studio .NET 2003, open an existing BizTalk Server project.
   
3. In the BizTalk Server project, click View, and then click BizTalk Explorer.
   
4. In BizTalk Explorer, expand BizTalkMgmtDb.dbo, expand Receive Ports, expand the specific receive port for the project, expand Receive Location, expand the specific receive location for the project, right-click the receive location, and then click Edit.
   
5. In the Receive Location Properties - Configuration - General dialog box, for Address URI, click the ellipsis (...) button.
   
6. In the FILE Transport Properties dialog box, set the BatchSize value to 1, and then click OK. By default, this value is set to 20 for the FILE adapter.
   

Use the following procedure to create a registry key that specifies the messaging thread pool size. This value specifies a limit to the number of threads that a single processor can spawn.

To specify the messaging thread pool size

1. Click Start, and then click Run.
   
2. In the Run dialog box, in the Open box, type regedit, and then click OK.
   
3. In Registry Editor, expand HKEY_LOCAL_MACHINE, expand SYSTEM, expand CurrentControlSet, expand Services, right-click BTSSvc*, point to New, click DWORD Value, type MessagingThreadPoolSize, and then press ENTER.
   
4. In Registry Editor, double-click MessagingThreadPoolSize.
   
5. In the Edit DWORD Value dialog box, in the Value data box, type 1, and then click OK.
   

To configure tracking, BizTalk Server 2004 provides various settings for the tracking service, tracking interceptors, the Health and Activity Tracking (HAT) tool, and Business Activity Monitoring (BAM). The following table shows the various tracking configurations that were set for all of the tests in the tracking section.

Table 34 Tracking configurations set for the tracking tests

By default, BizTalk Server 2004 enables the tracking service and tracking interceptors. You can disable the tracking service and/or interceptors through the BizTalk Server Administration tool, or the BizTalk Server Windows Management Instrumentation (WMI) interfaces.

When the tracking service is disabled, BizTalk Server stops moving tracking data from the MessageBox database to the Tracking database. With the tracking interceptors disabled, BizTalk Server stops collecting tracking data in the message box.

To enable or disable the tracking of pipeline components and message properties, use the HAT configuration menu. BizTalk Server automatically enables BAM tracking after you define business activities and associate them with orchestrations.

To test the performance impact of tracking in BizTalk Server 2004 with a single message box, two scenarios were used: messaging and orchestration. This section describes each scenario.

Messaging Scenario with One Message Box

Orchestration Scenario with One Message Box

In this orchestration scenario, BizTalk Server 2004 uses a basic orchestration to determine whether inbound requests should be accepted or rejected. The determination is made based on one field of the request, and a response is sent accordingly. The orchestration scenario uses the same pipeline components as the previous messaging scenario. For each document that it receives, BizTalk Server processes one document and sends one document. The following figure shows the data flow diagram for the performance testing of BizTalk Server 2004 tracking in an orchestration scenario with one message box.

Figure 30 Data flow diagram for orchestration with one message box

This section shows the test results for the messaging and orchestration scenarios. The following table shows the test results for the messaging scenario.

Table 35 Test results for the messaging scenario

The bottleneck for this scenario and deployment configuration was the CPU processor utilization on the orchestration server. The throughput statistics were obtained by maximizing the CPU processor utilization on the orchestration server. You can alleviate this bottleneck by adding servers to distribute the workload. The following table shows the test results for the orchestration scenario.

Table 36 Test results for the orchestration scenario

For the scenarios above, all tracking configurations had similar performance except the configuration where tracking was disabled. The orchestration scenario provided better performance than the messaging scenario because the messaging scenario involved transformation maps and processed two documents for each document received (the orchestration scenario sent one document for each document received). The orchestration scenario also included an additional server dedicated to processing orchestrations, thereby increasing the capacity of BizTalk Server to receive, process, and send documents more efficiently.

In the messaging scenario, the default tracking configuration had minimal impact on throughput. Throughput increased by 6 percent when tracking was off, and by 13 percent when the tracking service and interceptors were off. Message-body tracking in the HAT 1 configuration (where tracking is on for the receive outbound pipeline and send inbound pipeline) degraded performance by less than 2 percent over the default tracking configuration. The default tracking configuration provided 36 percent and 31 percent more throughput than the HAT 2 and 3 configurations, respectively.

In the orchestration scenario, throughput increased by 17 percent when the tracking service and interceptors were off. The default tracking configuration provided up to 15 percent more throughput than message-body tracking in the HAT 1 configuration. The default configuration also provided 2 percent, 8 percent, and 12 percent more throughput in the HAT 1, 2, and 3 configurations, respectively. With HAT off, the default configuration provided 5 percent more throughput than with BAM tracking on, and 23 percent more than HAT message-body tracking with BAM on.

When tracking is off, BizTalk Server 2004 can process 516 documents per second at an incoming rate of 258 received documents per second (test duration was over 27 hours). When the incoming rate is increased, processing speed decreases but publishing rate increases. Improving overall performance requires finding the right balance between processing and publishing rates, and minimizing message accumulation in the application queue.

In the second series of tests where tracking is on and the Tracking database is archived, file reception was decreased to 181 to 184 documents per second to achieve a sustainable amount of workload, and the test duration lasted over 17 hours. If consistent load persists for longer periods of time, BizTalk Server encounters performance degradation until the load is alleviated. Performance degradation is most noticeable when processing and publishing speeds far exceed the rate at which the tracking service moves tracking data. For this configuration, the tracking service can move 441 messages per second.

In the third series of tests where tracking is on and the Tracking database is not archived, test duration lasted 10 hours at an incoming rate of 181 to 184 documents received per second. For this configuration, the tracking service can move 119 messages per second, and total average throughput over 17 hours was 260 documents processed per second.

The policy used in this test contains rules like the following:

If ( ClassN.Attribute >= constant ) 
Then {Do something}

Note
Different .NET classes were used for the rules in the policy.

The application used to run the test executes the policy from four different threads. The average CPU utilization on the BizTalk Server hosting the Business Rule Engine is approximately 80 percent. The following graph shows the results of the test.

Figure 31 Business rules test results for .NET classes

The policy used in this test contains rules like the following:

If (XmlDocumentMemberN.Field >= constant ) 
Then { Do something}

The application used to run the test executes the policy from four different threads. The average CPU utilization on the BizTalk Server hosting the Business Rule Engine is approximately 90 percent. The following graph shows the results of the test.

Figure 32 Business rules test results for XML documents

The Business Rule Engine supports the following three database-related types:

• TypedDataRow. Constructed by using a reference to an ADO.NET DataRow instance. The TypedDataRow is an obvious choice for rules that deal only with data from one or a small number of rows from a particular table.
  
• TypedDataTable. Literally a collection of TypedDataRow objects. Each row in the database table is wrapped as a TypedDataRow and asserted into the working memory by the rule engine.
  A TypedDataTable requires an in-memory ADO.NET DataTable, which can be a performance overhead if this particular DataTable contains a very large number of rows. If a small number of rows in the database table are relevant and you can determine these rows before calling the rules, use a DataTable; otherwise use TypedDataRow. The assumption is that a high number of rows in the DataTable are relevant to the rules.
  
• DataConnection. Represents a table in a database accessed through a database connection.
  The difference between DataConnection and TypedDataTable is that in addition to the dataset name and table name, DataConnection requires a usable database connection and optionally a database transaction context.
  

The test scenarios included in this section capture the performance differences between data connection and data table bindings and show which of these bindings is more efficient depending on the scenario.

Increasing the Number of Data Rows

This test used the following rule:

If (Table1.Column1 = 1) 
Then Class1.Property1 = Table1.Column1

The test started with a few rows in the test table and gradually increased the number of rows. The purpose of this test was to show how the data connection binding becomes more efficient than data table binding when the data set becomes larger. The following graph shows the results of the test.

Figure 33 Business rules test results for data connections and tables

Increasing the Number of Conditions

This test used a policy that contained 100 rules, and changed the number of unique conditions in the rules.

                        Rule N
If (Table1.Column1 = (N Mod NumberofUniqueConditions) 
Then Class1.Property1 = Table1.Column1

The purpose of the test was to show how the data connection binding becomes much more efficient when the rule set uses shared conditions/queries. The following graph shows the results of the test.

Figure 34 Business rules test results for increasing the number of conditions

Increasing Query Selectivity

For this test, multiple policies were used, and each policy contained only one rule:

If (Table1.Column1 <= query_value) 
Then Class1.Property1 = Table1.Column1

The purpose of this test was to show how data connection bindings are more efficient when the more restrictive queries/conditions are used. However, the performances of the data table and data connection bindings converge as the queries/conditions become less restrictive. The following graph shows the results of the test.

Figure 35 Business rules test results for increasing query selectivity

Consider the following guidelines when optimizing DataConnection bindings:

• Use primary keys. When there is a primary key, the equality of two rows is determined by whether the rows have the same primary key, rather than by object comparison. If the rows are determined to be the same, only one copy is retained in memory, and the other is released (resulting in less memory consumption).
  When a DataConnection is asserted into the rule engine for the first time, the engine always tries to locate its primary key information from its schema. If a primary key exists, primary key information is retrieved and used in all subsequent evaluations.
  
• Provide running transactions. Without a transaction, each query and update on the DataConnection will initiate its own local transaction, and different queries might return different results in different parts of rule evaluations. Users may experience inconsistent behavior if there are changes in the underlying database table.
  Although you can use a DataConnection without providing a transaction when the table does not change over time, it is recommended that a transaction be used even when the DataConnection is only being used for read operations.
  
• Use OR conditions with caution. If the rule uses only conjunctive (AND) conditions, tests and queries are executed as early as possible, so instances of objects passing through are reduced. As a result, the number of queries against the subsequent DataConnection is reduced proportionally. If disjunctive (OR) conditions and a DataConnection are used together in a rule, all condition evaluations are pushed to the final query. If more than one DataConnection is used in a rule, all queries except the last one effectively become a Select-ALL query statement.
  In general, it is better to split any rule with an OR condition into two or more discrete rules, because the use of OR conditions decreases performance compared to the definition of more atomic rules. This observation is true whether or not DataConnections are used.
  You might also consider using separate rules that consist only of conjunctive conditions instead of one rule with OR conditions. With OR conditions, the number of queries grows at the speed of multiplication of instances of all joining objects. This is shown in the following example:
  
  IF (A.x == 7 OR A.x == 8) AND DC.y == A.y 
  THEN DC.z =10

  In this example, A represents an ObjectBinding; DC represents a DataConnection, and x, y, and z represent attributes of A and DC. If A has 100 instances, and x is 1 in the first object, 2 in the second object, through 100 in the 100th object, 100 queries have to run against the DataConnection.
  It is better to rewrite the preceding rule by splitting it into two rules:
  
                          Rule 1 
  IF A.x == 7 AND DC.y==A.y
  THEN DC.z =10
  Rule 2
  IF A.x == 8 AND DC.y==A.y
  THEN DC.z =10

• Use DataConnection instead of TypedDataTable. In many scenarios, using DataConnection provides better performance and consumes less memory than using TypedDataTable. However, TypedDataTable may be required in some cases because of certain restrictions on using DataConnection. In some other cases, using TypedDataTable may yield better performance than using DataConnection.
  Use TypedDataTable instead of DataConnection under the following conditions:
  
  • Data changes need to be made but the table does not have a primary key. To make data changes by using DataConnection, a primary key is required. Therefore, if there is no primary key, TypedDataTable is the only viable approach.
    
  • Selectivity is high, which means that a large percentage of rows in the table will pass the tests specified as rule conditions. In this case, DataConnection does not provide much benefit and it may perform worse than TypedDataTable.
    
  • The table is small—typically, a table that contains fewer than 500 rows. Note that this number could be larger or smaller depending on the rule shape and on the memory available to the rule engine.
    
  • Rule-chaining behavior is expected in the policy. Calling the Update function on a DataConnection is not supported, but you could invoke DataConnection.Update in a rule using a helper method. When rule chaining is required, TypedDataTable is a better choice.
    
  • One or more columns in the table hold a very large amount of data that is not required by the rules. An example is an image database, where the columns hold the image (large amount of data), name, date, and so on. If the image is not required, it may be better to select only the columns needed by the rules. For example, issuing a query such as "SELECT Name, Date from TABLE" can be more efficient than using DataConnection.
    
  • If many rules need or update the same database row, using a TypedDataTable, the row is shared between all rules, and if the condition is the same (for example, Table.Column == 5), the condition evaluation can be optimized. With a DataConnection, in general, a query is generated for each rule that uses the DataConnection. Although the rows are reused (if the table has a primary key), multiple queries could be generated to get the same data each time.