Export (0) Print
Expand All
Expand Minimize

Performance Comparison: Exposing Existing Code as a Web Service

 

Priya Dhawan
Microsoft Developer Network

October 2001

Summary: This article provides a performance comparison between three Web Service implementations that expose existing COM components and Transact SQL code and an equivalent Web Service implementation using ASP.NET. (24 printed pages)

January 2005 Update: SOAP Toolkit 2.0 is no longer supported. Information in this article may apply to SOAP Toolkit 3.0, which has been deprecated by the Microsoft .NET Framework. See the SOAP Toolkit page on the MSDN Web Services Developer Center for information on available support for SOAP Toolkit 3.0.

Contents

Introduction
Architectural Options
Test Scenarios
Test Tool
Test Machines Configuration
Test Strategy
Performance Test Results
Conclusion

Introduction

In the previous three articles on Exposing Existing Code as a Web Service, we discussed ways of exposing existing COM components and Transact SQL code as Web Services using Microsoft® .NET Framework, Microsoft® SOAP Toolkit 2.0, and Microsoft® SQL Server™ 2000. In this article, we will compare the performance of these three implementations of the Web Service with a simple Web Service implemented using ASP.NET.

Architectural Options

Though externally similar, the different versions of our Web Service have been implemented using different technologies and, hence, have different underlying architectures. For example, they use different data access techniques and objects to retrieve data from the database.

ASP.NET Web Service

This version of our Web Service (we will refer to it as WS0-Managed in this article) is an ASP.NET Web Service implemented using Microsoft Visual Basic .NET. The Web Service uses ADO.NET to access the database. The .NET data types, simple as well as complex, are transparently serialized and deserialized as XML by the .NET Framework, relieving you of the extra work of transforming complex data types into XML.

Because the performance comparison is primarily between the Web Services that expose existing code, we used this version merely as a control to base our test results against and implemented it using simple rather than optimized code. We'll compare the performance of an optimized version of this Web Service with other Web Services in an upcoming article.

ms978401.bdadotnetarch11_01(en-us,MSDN.10).gif

Figure 1. ASP.NET Web Service

ASP.NET Web Service Exposing Existing COM Component through Interop

This Web Service (we'll refer to it as WS1-Interop in this article), implemented using ASP.NET, wraps an existing Visual Basic 6.0 COM component and handles serializing and deserializing complex data types such as the ADO Command object, ADO Recordset, etc., exposed by the existing code as XML. To interoperate with a COM component, the .NET Framework creates a run time callable wrapper (RCW), which is essentially a proxy to the COM component that marshals calls between the .NET application and the COM component. Although run time callable wrappers provide interoperability with COM components, they do introduce an extra layer to traverse before you access the COM component itself. Each and every call being made from the Web Service to the COM component goes through the COM interoperability layer.

ms978401.bdadotnetarch11_02(en-us,MSDN.10).gif

Figure 2. ASP.NET Web Service through COM Interop

As you see in Figure 2, the Visual Basic 6.0 COM component is registered in a COM+ library package. The reason is that exposing an apartment model object through an ASP.NET Web Service is not supported unless the apartment model object is registered with the COM+ services. It can be registered either in a library package or a server package. Be sure to mark "Requires Synchronization" in case it is a library package residing on Microsoft Windows® 2000 Service Pack 2 or later. You will not achieve good performance if you run your apartment model object outside the COM+ services. For a more detailed description of this implementation see Exposing Existing Code as a Web Service Using the .NET Framework.

SOAP Toolkit 2.0 Web Service Exposing Existing COM Component

This Web Service (we'll refer to it as WS2-STK2 in this article), implemented using SOAP Toolkit 2.0, makes use of a custom type mapper to transform complex data types exposed by the existing Visual Basic 6.0 COM component into XML. A custom type mapper provides a mechanism to easily deal with complex types in your Simple Object Access Protocol (SOAP) application. SOAP Toolkit 2.0 passes the incoming SOAP request message as an IXMLDomNode object to the custom type mapper, which in turn deserializes it as a valid COM data type with which your application can work. A custom type mapper also turns COM data types exposed by your application into a valid XML representation to be sent in a SOAP message. For more detail on this implementation see Exposing Existing Code as a Web Service Using the SOAP Toolkit 2.0.

ms978401.bdadotnetarch11_03(en-us,MSDN.10).gif

Figure 3. SOAP Toolkit 2.0 Web Service

SQL Server 2000 Web Service

The XML features of SQL Server 2000 can be used to expose existing Transact SQL code as a Web Service. You can define a virtual root on a Microsoft Internet Information Services (IIS) server, which gives you HTTP access to the data and XML functionality of SQL Server 2000. This Web Service (we'll refer to it as WS3-SQL in this article) utilizes the XML feature of SQL Server to expose SQL stored procedures as Web Services. This Web Service uses template files, which are well-formed XML documents containing one or more SQL statements and XPath queries, to invoke stored procedures on the database server and XSL style sheets to transform outgoing XML into SOAP messages. More on this implementation is covered in Exposing Existing Code as a Web Service Using SQL Server 2000.

ms978401.bdadotnetarch11_04(en-us,MSDN.10).gif

Figure 4. SQL Server 2000 Web Service

Test Scenarios

These Web Services perform operations that are common in business applications such as getting a customer record, getting customer orders, inserting a customer, etc. These operations are implemented as Web Service methods with similar code for all the versions of the Web Service. We'll see how the performances of these Web Services match up later in this article.

The cost of an operation is affected by many factors. An operation fetching a single record from a database performs considerably faster than the one that fetches thousands of rows. One reason is the fact that the transmission time of the data varies with the size of the data being returned.

The size of the database itself is another factor. Depending on the database architecture and the type of queries being performed, the amount of data may affect the cost of data operations.

The number of simultaneous users sending requests to your application also significantly affects performance. An application could be very responsive to a small number of simultaneous users, but suffer severe performance degradation as the number of simultaneous users increases. The number of simultaneous users being served up by the application before the performance starts degrading gives a measure of application scalability. A scalable application might pay a performance penalty for using resources efficiently, but is prepared to handle multiple concurrent users.

To make the test as realistic as possible, the database was loaded with over 100,000 rows of Customer accounts, 1 million rows of Orders (10 orders per Customer), and over 5 millions rows of Order details (5 details per Order).

Next we'll go over the methods that we used to compare the Web Services.

GetOrderStatus

The GetOrderStatus method accepts an OrderId and returns an integer representing the status of the order.

GetCustomer

The GetCustomer method accepts a CustomerId and returns a single row containing information about the customer.

GetCustomers

The GetCustomers method accepts a CustomerId and a parameter to specify the number of customer records you want to read. The top n records with CustomerId greater than the CustomerId passed to the Web Service method are read.

We performed tests with paging through a large set of customer records with different chunk sizes: 100, 500, and 1000.

GetOrders

The GetOrders method retrieves a hierarchy of orders and their details from the database. This method accepts an OrderId and a parameter to specify the number of orders you want to read. The top n records with OrderId greater than the OrderId passed to the Web Service method and all the corresponding details are read.

We performed tests by paging through a large set of orders and their details with different chunk sizes of 10 orders and 50 details, 50 orders and 250 details, 100 orders and 500 details, and 200 orders and 1000 details.

InsertCustomer

This InsertCustomer method inserts a Customer record into the database and returns a CustomerId as an integer. This method accepts all the required parameters about the customer record as inputs and invokes a stored procedure that does the insertion and returns the CustomerId of the record just inserted.

InsertOrder

In the InsertOrder method we insert a hierarchy of an order and its details into the database. This method accepts an XML string representing the order and its details and returns the OrderId as an integer.

The tests were performed by inserting a single order header and varying number of details. The data set that we used was 1 Order and 2 Details, 1 Order and 10 Details, and 1 Order and 20 Details.

Test Tool

There are a number of technologies that can be used to create test scripts from scratch—including Microsoft Visual Basic® Scripting Edition (VBScript), Microsoft JScript®, and Perl. However, there are also commercial products available that provide frameworks for developing automated tests. Each product has strengths with respect to different types of software development projects.

For the purpose of our tests, we used Application Center Test, which is a component of Microsoft Visual Studio® .NET and is designed to stress test Web servers and analyze performance and scalability problems with Web applications, including ASP pages and the components they use. It can simulate a large group of users by opening multiple connections to the server and rapidly sending HTTP requests.

Application Center Test also allows us to build realistic test scenarios where we can call the same Web Service method with a randomized set of parameter values. This is an important feature whereby users are not expected to call the same Web Service method with the same parameter values over and over again.

Test Machines Configuration

The following tables provide a brief summary of the test bed configuration used to perform the tests.

Table 1. Client Machine Configuration

Number Of clientsMachine/CPU# Of CPUMemoryDiskSoftware
1Compaq ProLiant DL360 1000 MHz2128 MB30 GB
  • Windows 2000 Server SP 2
  • Visual Studio .NET Beta 2

Table 2. Web Server Configuration

Number Of serversMachine/CPU# Of CPUMemoryDiskSoftware
1Compaq ProLiant DL580 702 MHz41 GB16.95 GB
  • Windows 2000 Advance Server SP 2
  • .NET Framework Beta 2
  • Visual Studio .NET Beta 2

Table 3. Database Server Configuration

Number Of serversMachine/CPU# Of CPUMemoryDiskSoftware
1Compaq ProLiant DL580 702 MHz41 GB16.95 GB
  • Windows 2000 Advance Server SP 2
  • SQL Server Enterprise Edition SP 2

Test Strategy

An ASP.NET Web Service automatically supports clients using the SOAP, HTTP-GET, and HTTP-POST protocols to invoke Web Service methods. SOAP Toolkit 2.0 supports only the SOAP protocol, whereas the SQL XML interface allows you to query the database using HTTP-GET and HTTP-POST. For a fair comparison between the Web Services, we used SOAP as the primary transport and serialization protocol for the Web Services. To support SOAP with SQL XML interface, we parse incoming SOAP messages using the sp_xml_preparedocument system procedure of SQL Server 2000 and transform outgoing XML into SOAP messages using XSL style sheets.

For these tests, we used a single client machine, one Web server, and one database server. While running tests, we found that at high user load, the CPU on the Web server achieved ~100 percent utilization, which impaired performance of the Web Service. If the Web server becomes the performance bottleneck at high user load, you can achieve better results if you load balance incoming user requests to two or more Web servers.

The client load was generated using Application Center Test, as discussed earlier. We developed test scripts for every method of the Web Services. We tried to randomize the requests to get a different Customer or Order, depending on the method being executed from within the test script. For example:

Dim conn, req, heads,head
Dim serverName
Dim UB, LB
' Set the upperbound for Orders list
UB = 1000000
' Set the lowerbound for Orders list
LB = 1
' Set the web server name
serverName = "appleweb01"
' Create a new Connection for the web server
Set conn = Test.CreateConnection(serverName)
If conn is nothing then
   Test.Trace("error")
else
' Create a new request
Set req = Test.CreateRequest
' Change the default HTTP Version to 1.0
req.HTTPVersion = "HTTP/1.0"
' Set the 
req.Path = "/BDAdotNetWebService4/BDAdotNetWebService4.asmx"
' Set the HTTP method type
req.Verb = "POST"
' Get the HTTP headers used in the request
Set heads = req.Headers
' Add a new header to the request
heads.Add "SOAPAction", "http://tempuri.org/Example1"
' Get the Content-Type header
set head = heads.item("Content-Type")
' Set the value of the Content-Type header
head.value = "text/xml; charset=utf-8"
' Use the Randomize funtion to initialize the Rnd function
Randomize
' Set the body of the HTTP POST request with the data 
req.Body = "<?xml version=""1.0"" encoding=""utf-8""?>" & _
  "<soap:Envelope " & _
  "xmlns:soap=""http://schemas.xmlsoap.org/soap/envelope/"" " & _ 
  "xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" " & _
  "xmlns:xsd=""http://www.w3.org/2001/XMLSchema"">" & _
  "<soap:Body>" & _
             "<Example1 xmlns=""http://tempuri.org/"">" & _
           "<OrderId>" & int((UB - LB + 1)*Rnd + 1 ) & "</OrderId>" & _
            "</Example1>" & _
             "</soap:Body>" & _
  "</soap:Envelope>"
' Send the request
conn.Send req
End if

Note that we set the HTTP version to 1.0 instead of using the default, which is 1.1. The reason for this is that a Web Service runs into Nagling issues if the response is small. Nagling prevents Transmission Control Protocol (TCP) from sending small packets by adding a delay after each buffer is received from the Internet Server Application Programming Interface (ISAPI) on a particular connection in case another buffer is coming soon so the two can be combined. In realistic benchmarks with thousands of clients hitting the server in parallel, Nagling will often help performance by reducing the number of small packets sent. In general, if your benchmarks are giving much worse results than you think you should be getting, try turning off Nagling to see if that helps.

To turn off the Nagling delay in SOAP Toolkit 2.0, change the "NoNagling" DWORD value in HKEY_LOCAL_MACHINE\Software\Microsoft\MSSOAP\SOAPISAP to one (1).

The recommended means of turning off Nagling delay in .NET Beta 2 is to turn off Connection:keep-alive. Though we can disable Connections:keep-alive with HTTP 1.1, IIS 5.0 tries to optimize in some cases and you would still experience the Nagling delays.

Performance Test Results

In this section we will discuss the performance statistics gathered while running the stress tests to compare the Web Services. We measured throughput of the Web Services for each of the methods we planned to test against. In general, throughput is the number of client requests processed within a certain unit of time, typically within a second. Because peak throughput may occur at a response time that is unacceptable from an operation usability standpoint, we capped the testing of a given method once the response time exceeded five seconds.

GetOrderStatus

Here we compare the behavior of Web Services when getting a single value from the database.

ms978401.bdadotnetarch11_05(en-us,MSDN.10).gif

Figure 5. GetOrderStatus performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • All the versions generate very similar XML containing the OrderStatus.

Figure 5 shows throughput as a function of simultaneous browser connections. It also depicts how many requests per second are processed at various user loads.

As demonstrated in the graph, the SQL XML implementation is considerably faster than other versions. At a user load of 100 concurrent users, this implementation running at peak throughput can serve up to 600 requests per second.

The ASP.NET implementation reaches its peak throughput of 492 requests per second at a user load of 50 concurrent users.

The SOAP Toolkit implementation can handle up to 50 concurrent users at a peak throughput of 422 requests per second.

The interop version achieves a throughput of 332 RPS at a user load of 100 concurrent users. Because calling a method on a COM component involves first instantiating the component, then calling the method on the COM component, and finally accessing the object returned by the component, WS1-Interop deals with multiple calls through the COM interoperability layer and in turn pays a small performance penalty.

GetCustomer

Here we compare the behavior of the Web Services when getting a single row from the database.

ms978401.bdadotnetarch11_06(en-us,MSDN.10).gif

Figure 6. GetCustomer (Customer=1) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • All the versions generate very similar XML representing a customer record.

As you see in Figure 6, the SQL XML implementation achieves its peak throughput at a user load of 20 concurrent users. The ASP.NET implementation is able to handle 100 concurrent users at its peak throughput of 492 RPS and the SOAP Toolkit implementation can serve up 50 concurrent users at its peak throughput of 411 RPS. The interop implementation achieves a throughput of 320 RPS at a user load of 100 concurrent users.

To explain the numbers that we see on the graph, we will need to go into the implementation details. WS3-SQL uses a SQL template to invoke the stored procedure, which returns the customer record as XML using the FOR XML clause in the SELECT statement. Because the output is already generated as XML, WS3-SQL only builds a SOAP message around the returned XML using an XSL style sheet.

WS0-Managed uses an ADO.NET SQL Command object to retrieve the customer record as its output parameters. The output parameters of the Command object are transformed into XML containing name and value pairs using the StringReader and XmlTextReader classes.

The existing COM component method returns an ADO Command object containing the customer record as its output parameters. WS2-STK2 uses a custom type mapper to transform the output parameters of the Command object into XML containing name and value pairs.

In addition to instantiating and calling the method on the existing COM component that returns an ADO Command object, WS1-Interop iterates through the parameters collection of the Command object to access all of the output parameters. Because accessing a property or a method of a COM component is a separate call through the COM interoperability layer, this Web Service method incurs some additional performance cost as compared to other versions.

GetCustomers

In this section we will compare the behavior of Web Services when fetching multiple records. We performed tests with a result set of 100 records, 500 records, and 1000 records. We will also observe how the size of data impacts performance.

ms978401.bdadotnetarch11_07(en-us,MSDN.10).gif

Figure 7. GetCustomers (Customers=100) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing customer records. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQl generates a well-formed XML representing customer records.

As you would expect, getting more records from the database impacts performance. If you compare Figure 7 with Figure 6, you will notice a decrease in the throughput of the Web Services. Besides retuning more data, we are also dealing with different data access objects to retrieve the result. The WS3-SQL implementation remains the same. In the previous method, we used the Command object to retrieve a single row as its output parameters. In this case, the existing COM component returns the Recordset object, which is transformed to XML using ADO Stream as an intermediate object. The WS0-Managed Web Service populates a Dataset object with the result, which transparently gets transformed into XML.

Figure 7 shows that the WS3-SQL Web Service can handle requests from 20 concurrent users and produce approximately 195 requests per second. The WS2-STK2 achieves its throughput of 185 requests per second at a user load of 100 concurrent users. The WS0-Managed Web Service is capable of servicing up to 100 users when the throughput levels off at 145 requests per second.

Even though the WS1-Interop Web Service involves a lot of COM calls through the COM interoperability layer, it is able to handle 50 concurrent users at a throughput of 156 RPS. Besides instantiating the COM component and calling the appropriate method on it, WS1-Interop transforms the ADO Recordset object returned by the COM component into XML. To transform the Recordset object into XML, the Web Service method creates an ADO Stream object, saves the XML representation of the Recordset object into the Stream object, and then reads the XML string using the ReadText method of the Stream object.

ms978401.bdadotnetarch11_08(en-us,MSDN.10).gif

Figure 8. GetCustomers (Customers=500) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing customer records. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQl generates a well-formed XML representing customer records.

The performance further drops when we increased the number of records returned by the database to 500. Compare Figure 8 with the previous two graphs, where we fetched a single customer and then 100 customers.

WS3-SQL is capable of servicing 20 concurrent users at a throughput of 100 requests per second, after which it levels off. WS2-STK2 achieves its peak of 69 requests per second at a user load of 20 concurrent users. WS0-Managed behaves very similar to WS2, peaking at 66 requests per second at a user load of 20 concurrent users.

Despite going through the COM interoperability layer for every COM call it makes, WS1-Interop behaves very similar to WS2-STK2. It peaks at 68 RPS at a user load of 20 concurrent users.

ms978401.bdadotnetarch11_09(en-us,MSDN.10).gif

Figure 9. GetCustomers (Customers=1000) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing customer records. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQl generates a well-formed XML representing customer records.

The performance further drops when we increased the number of records returned by the database to 1000. WS3-SQL achieves 55 requests per second at a user load of 20 concurrent users. WS2-STK2 is able to handle 20 concurrent users at 38 requests per second. WS1-Interop is able to handle 10 concurrent users at 34 RPS and WS0-Managed supports 20 concurrent users at 30 requests per second.

GetOrders

In this section we will compare the behavior of Web Services when fetching a hierarchical result set. We performed tests with a result set of 10 orders and 50 details, 50 orders and 250 details, 100 orders and 200 details, and 200 orders and 1000 details. We will also observe how the size of data impacts performance.

ms978401.bdadotnetarch11_10(en-us,MSDN.10).gif

Figure 10. GetOrders (Orders=10, Details=50) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing orders. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQL generates a well-formed XML representing orders.

We started with a result set of 10 orders and 50 details. As shown in Figure 10, WS3-SQL achieves the throughput peak of 186 RPS at a user load of 20 concurrent users. After that it levels off. WS0-Managed can handle up to 50 concurrent users at 135 RPS before it saturates, whereas WS2-STK2 evens out at around 125 RPS at a user load of 50 concurrent users. WS1-Interop achieves 120 RPS at a user load of 50 concurrent users.

One of the facts that can be attributed to the difference in performance numbers is that these versions are using different methods to build the hierarchy between orders and details. Another factor is the data object being used to populate the data. WS3-SQL uses the FOR XML clause with EXPLICIT in the SELECT statement to shape the hierarchy.

WS0-Managed sets the relationship between the two tables containing orders and details in the data set. Once you set the relationship, the .NET Framework automatically generates the XML representation of the data set maintaining hierarchy between orders and their details.

WS2-STK2 uses the MSDATASHAPE provider to define the hierarchy. Because the existing COM component returns an ADO Recordset object populated with orders and details, WS2-STK2 calls a custom type mapper to transform the recordset into XML. An ADO Stream object is used as an intermediate object to save the XML representation of the recordset.

WS1-Interop deals with transforming an ADO Recordset object into XML using the ADO Stream object, which overall turns into a lot of COM calls.

ms978401.bdadotnetarch11_11(en-us,MSDN.10).gif

Figure 11. GetOrders (Orders=50, Details=250) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing orders. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQL generates a well-formed XML representing orders.

Figure 11 displays the performance results when retrieving a result set of 50 orders and 250 details from the database. Because we have increased the amount of data being returned from the database, we see a decrease in performance of all the Web Services, as compared to what we got in the previous method where we fetched only 10 orders and 50 details.

WS3-SQL can manage up to 100 RPS at a user load of 20 users. WS0-Managed supports the same number of users but at a slighter throughput of 80 RPS.

WS1-Interop is able to handle a user load of 20 concurrent users at 69 RPS.

WS2-STK2 evens out at 62 RPS supporting less than 20 concurrent users.

ms978401.bdadotnetarch11_12(en-us,MSDN.10).gif

Figure 12. GetOrders (Orders=100, Details=500) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing orders. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQL generates a well-formed XML representing orders.

Next we tested against a result set of 100 orders and 500 details. Because we have increased the amount of data being returned from the database, we see a decrease in performance of all the Web Services as compared to what we got in the previous methods.

WS3-SQL can manage up to 57 RPS at a user load of 20 users. WS0-Managed supports the same number of users but at a slighter throughput of 42 RPS. WS1-Interop behaves similar to WS0-Managed with 40 RPS at a user load of 20 users.

WS2-STK2 evens out at 37 RPS supporting 10 concurrent users.

ms978401.bdadotnetarch11_13(en-us,MSDN.10).gif

Figure 13. GetOrders (Orders=200, Details=1000) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • There are variations in the output XML generated by the Web Services. WS0-Managed generates the XML representation of a data set containing orders. WS1-Interop and WS2-STK2 generate the XML representation of a recordset. WS3-SQL generates a well-formed XML representing orders.

Next we tested against a result set of 200 orders and 1000 details. Because we have increased the amount of data being returned from the database, we see a decrease in performance of all the Web Services as compared to what we got in the previous methods.

WS3-SQL can manage up to 30 RPS at a user load of 10 users. WS1-Interop is able to handle 10 concurrent users at 21 RPS. WS0-Managed supports the same number of users but at a slighter throughput of 20 RPS.

WS2-STK2 evens out at 20 RPS supporting less than 5 concurrent users.

InsertCustomer

After getting an idea how these Web Services behave when dealing with reading a single value, a single row, multiple rows, and hierarchical data, we will move on to compare these Web Services when dealing with writing data the database.

Here we inserted a single customer in the database.

ms978401.bdadotnetarch11_14(en-us,MSDN.10).gif

Figure 14. InsertCustomer (Customer=1) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • All versions generate very similar XML representing the CustomerId of a newly inserted customer.

WS3-SQL again performs better than the other versions. It evens out at 472 RPS and can support 100 users. WS0-Managed reaches up to 458 RPS but can handle only 50 users. WS2-STK2 can handle 50 simultaneous users at a peak throughput of 350 RPS.

WS1-Interop achieves 320 RPS at a user load of 20 concurrent users, after which it evens out.

A peek into the implementation detail will help us understand the number we see on Figure 14. WS3-SQL uses a SQL template to apply the incoming HTTP parameters to the parameters of the stored procedure to execute. The stored procedure inserts the customer record and returns the CustomerId as XML using the FOR XML clause, which is transformed into a SOAP message by applying an XSL style sheet.

WS0-Managed populates an ADO.NET SQL Command object with the input data about a customer and invokes the stored procedure that inserts the customer record. The stored procedure returns the CustomerId of the record just inserted, which is returned to the Web Service consumers.

The existing COM component uses the ADO Command object to execute the stored procedure, which inserts the customer record in the database. It returns an integer representing the CustomerId of the new record to the Web Service, which in turn returns it to its consumers.

WS2-STK2 does not need a custom type mapper in this case, because SOAP Toolkit 2.0 handles the integer automatically.

InsertOrder

We conducted this last set of tests to see how inserting a hierarchy of an order and its details affects performance of the Web Services. We varied the number of detail rows to increase the amount of data being inserted.

ms978401.bdadotnetarch11_15(en-us,MSDN.10).gif

Figure 15. InsertOrder (Order=1, Details=2) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same existing COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • All versions generate very similar XML representing the OrderId of a newly inserted order.

If you compare Figure 15 with Figure 14, you will notice that WS3-SQL is now slightly slower than WS0-Managed and WS2-STK2. The reason could be that in this method the SQL Server itself has an extra overhead of parsing the SOAP message containing an order and its detail as a hierarchy. It also builds the XML representing the OrderId of the newly inserted order using the FOR XML clause. It achieves 375 RPS at a user load of 50 users, after which it evens out.

WS0-Managed is able to handle 100 users at ~390 RPS, after which the performance degrades. WS2-STK2 peaks at ~382 RPS with 50 users. WS1-Interop achieves 278 RPS at a user load of 50 users.

WS0-Managed and the COM component have a very similar implementation for this method. The former uses the ADO.NET Command object to execute the stored procedure, while the latter uses the ADO Command object. Both of them return an integer that represents the OrderId of the newly inserted order.

ms978401.bdadotnetarch11_16(en-us,MSDN.10).gif

Figure 16. InsertOrder (Order=1, Details=10) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • All versions generate very similar XML representing the OrderId of a newly inserted order.

Next we tested with inserting 1 order and its 10 details. Figure 16 shows that WS2-STK2 peaks at 301 RPS with around 100 users. WS0-Managed is able to handle 100 users at 260 RPS. WS3-SQL achieves 248 RPS at 50 users, after which it evens out. WS1-Interop levels off at 215 RPS with around 50 concurrent users.

ms978401.bdadotnetarch11_17(en-us,MSDN.10).gif

Figure 17. InsertOrder (Order=1, Details=20) performance comparison

Notes:

  • WS1-Interop and WS2-STK2 use the same existing COM component.
  • WS0-Managed, WS1-Interop, and WS2-STK2 use the same SQL stored procedure.
  • All versions generate very similar XML representing the OrderId of a newly inserted order.

Next we tested with inserting 1 order and its 20 details. Figure 17 shows that WS2-STK2 peaks at 185 RPS with around 50 users. WS0-Managed is able to handle 20 users at 165 RPS after which it evens out. WS3-SQL achieves 130 RPS at 20 concurrent users.

Conclusion

The best way to expose existing code as a Web Service depends on several factors. All the Web Services covered in this article have their advantages and disadvantages, and therefore are suitable for different purposes.

WS0-Managed: As we showed in this article, WS0-Managed exhibited pretty good performance with all the operations we tested. The .NET Framework in general offers a lot of advantages. One of them is that the code is easy to maintain and very easy to deploy. It offers good tool support for debugging purposes. Other features include output-caching, support for both synchronous and asynchronous communication between the client and the Web Service server, and support for different transport protocols such as SOAP, HTTP-GET, and HTTP-POST.

As mentioned earlier, the performance comparison is primarily between the Web Services that expose existing code, and therefore we implemented this version using the simplest code rather than an optimized one. We can use highly optimized objects such as DataReader offered by the .NET Framework to get even better performance. We'll compare the performance of an optimized version of this Web Service with other Web Services in an upcoming article.

WS1-Interop: Despite involving a lot of COM calls to process a request, this Web Service exhibited good performance and was comparable to other versions. The only restriction it poses is that you have to register the apartment model component in a COM+ application when exposing it through .NET Web Services. As already mentioned, you will get poor performance if your apartment model component is not registered in a COM+ application. Also, maintenance and debugging is not as simple as that provided by the .NET Framework.

WS2-STK2: This version demonstrated good performance in handling complex interfaces exposed by the existing code. However, the SOAP Toolkit has a couple of limitations. First, it only supports SOAP as the transport protocol, so the client must also support SOAP to communicate with the Web Service. Maintenance and debugging is not as simple as that provided by the .NET Framework.

WS3-SQL: This version exhibited very good performance in almost all the test scenarios. Although it provides good performance, it is the hardest to code, and because the Web Service involves queries with the FOR XML clause, it could get even more complicated if the data needs to be presented as a deep hierarchy. Its performance declines with writes due to parsing of XML provided as input parameters. It can also put extra work on the database server itself, which may already be your main processor bottleneck. It is usually not a good choice to put more stress on your database server when that work can be distributed to a less expensive middle-tier server. Also, maintenance and debugging is not as simple as that provided by the .NET Framework.

Show:
© 2014 Microsoft