Doculabs Web Services Benchmark


Greg Leake
Microsoft Corporation

James Duff
Vertigo Software, Inc.

Applies to:
    Microsoft® .NET Framework 1.0 and 1.1
    Microsoft® Windows® 2000 and Windows Server™ 2003
    Microsoft® Internet Information Services 5.0 and 6.0
    Microsoft® Oracle® Managed Provider

Summary: In April 2003, Doculabs, an independent research and consulting firm, conducted a performance study of Web services, including the scalability and interoperability of various Web services frameworks. Microsoft Corporation and many J2EE application server vendors were invited to participate, but only Microsoft accepted the invitation to compete. In order to provide a useful performance comparison, Doculabs selected three Web services vendors and implemented the benchmark using the vendors' publicly available guidelines. The full set of results for the benchmark can be found on the Doculabs Web site. The goals of this document are to discuss how the Microsoft .NET application was constructed and tuned, and how the .NET Framework was used to integrate with JBoss, one of the disclosed vendors in the benchmark. (21 printed pages)

Download the Doculabs Bench Report Web Services Performance Benchmark Study.
Size: 2.0 MB
Format: PDF
Get the Adobe Acrobat Reader


Benchmark Specification
Testing Methodology
Microsoft .NET Implementation
Results Summary

Benchmark Specification

The Doculabs benchmark consisted of three business methods that needed to be exposed as Web services. These methods are described in Table 1:

Table 1. Business methods

Test MethodFunctional Requirement
Get Order DetailsFor a given order number, return the order header information, including customer data, along with the order line items for the order.
Get Customer DetailsFor a given customer ID, return a complex object containing all the details for a customer, including the billing and shipping addresses.
Insert CustomerAccept a well-formed customer object, and then insert the data into the database. The function returns the new customer ID if successful, or minus one if the method fails.

The goals of these tests was to measure the ability of each Web service framework to serialize and de-serialize data, using varying data set sizes, along with database transactions that both read and insert data. The mix of request types matches the types of transactions that might be seen in a typical Web-based application. The benchmark required that each framework used Oracle for the back-end database.

Each of the benchmark implementations was built using the same application model with the application split into three tiers. The first tier comprised a set of data access classes used to encapsulate all the data access logic. The second tier comprised a set of business tier classes where any business validation or processing would occur. Due to the simplicity of the benchmark, the business logic classes were quite simple and only needed to call into the data access classes. None of the business logic classes were allowed to reference any vendor-specific database libraries; they could only use the data access classes created as part of the application. The third tier comprised a Web services façade to hide the implementation details of the business-logic tier from the consumers of the Web services. The goal of this n-tiered approach was to provide an application that would be flexible to future changes in application design and requirements.

Testing Methodology

Load Runner 7.5 from Mercury Interactive was used to stress each of the Web service frameworks. The Doculabs tests where broken into two main sections. The first section was designed to stress the Web service host and see how the server was able to handle a large number of concurrent clients. The second test was designed to stress the Web service proxy classes and the ability of the proxy to funnel a large number of requests to a single Web service.


Figure 1: Web service host stress infrastructure

Figure 1 shows how the Load Runner clients are used to stress the Web service host. A SOAP trace tool, TcpTrace, was used to record the SOAP messages sent between a SOAP proxy and a Web service. In each case, the proxy and the Web service were built with the same framework. A Load Runner Script was created using the recorded SOAP calls. Using Load Runner's parameterization routines, the data in the SOAP envelope was randomized so that each test client could request any one of the one million orders or customers in the database.


Figure 2: Web services proxy stress infrastructure

Figure 2 shows the methodology used to test the proxy classes. ASP.NET and JSP pages were created to invoke each of the methods in the C# Web service proxy and Java proxies respectively. Each page was designed to read a parameter from the query string and use the value to create valid parameters that could be passed to the Web service proxy object. This enabled the Web services to be used to request any of the data records from the database at random.

The interoperability testing was structured in an identical way, except that the Web services proxy generation tools were used to create a proxy for a Web service, which was created with another vendor's framework.

Microsoft .NET Implementation

Application Architecture


Figure 3: .NET application architecture

Figure 3 shows the architecture for the .NET implementation of the Web services benchmark specification. For the .NET implementation, C# was the language chosen for all tiers of the application; however Visual Basic .NET could have been used for any or all the tiers in the application. In .NET, Web service façades are typically built using an asmx page as these allow for the simple exposure of methods using tags in the application code. By simply adding the [WebMethod] attribute in front of an asmx method a business function can be exposed as a Web service. IIS (Microsoft® Internet Information Services) is used to host the Web service pages. All that needs to be done is to create a virtual directory in IIS based on the folder where the asmx file resides on the server's file system. The C# code below is the code for the .NET Web service.

public Order GetOrderDetails(int orderId){
   OrderBL orderBL = new OrderBL();
   return orderBL.GetOrderDetails(orderId);

public Customer GetCustomer(int customerId){
   CustomerBL customerBL = new CustomerBL();
   return customerBL.GetCustomer(customerId);

public int InsertCustomer(Customer customerData){
   CustomerBL customerBL = new CustomerBL();
   return customerBL.InsertCustomer(customerData);

The business logic for the application was written in C# and was used to call data access classes also written in C#. The data access classes made calls to the ADO.NET framework libraries. To access the database tables and run SQL queries against the database, prepared statements were used, as they provide a secure mechanism to make database calls while offering similar performance to stored procedures. Both the business logic classes and the data access classes were compiled into a single .NET assembly that was then copied to the bin directory in the Web services virtual directory folder.

Thin data classes were created to pass data between each of the three code layers.

using System;

namespace Doculabs.BusinessLogic{

   public class LineItem 
      private int orderId;
      private int itemId;
      private int productId;
      private string productDescription;
      private int orderQuantity;
      private float unitPrice;

      public LineItem(){

      public LineItem(int orderId, int itemId, int productId, string productDescription, int qty, float unitPrice){
         this.orderId = orderId;
         this.itemId = itemId;
         this.productId = productId;
         this.productDescription = productDescription;
         this.orderQuantity = qty;
         this.unitPrice = unitPrice;

      public int OrderId{
         get { return this.orderId; }
         set { this.orderId = value; }
      public int ItemId{
         get { return this.itemId; }
         set { this.itemId = value; }
      public int ProductId{
         get { return this.productId; }
         set { this.productId = value; }
      public string ProductDescription{
         get { return this.productDescription; }
         set { this.productDescription = value; }
      public int OrderQuantity{
         get { return this.orderQuantity; }
         set { this.orderQuantity = value; }
      public float UnitPrice{
         get { return this.unitPrice; }
         set { this.unitPrice = value; }

In order to confirm that the Web service worked before benchmarking the application, we used the Web service test page that ASP.NET creates for each WebMethod available on the asmx page. Figure 4 shows the test page, which allows you to invoke the Web service directly. The test page includes a sample SOAP message and the type of SOAP response it would generate.


Figure 4: Web service test page


The .NET tuning parameters were specified at two levels. The first level is the Web application, which is controlled by settings in the web.config configuration file. The second level of tuning occurred at the system level, and how this is done depends on the version of Windows that is being used. With Windows 2000, the ASP.NET process model is controlled by settings in machine.config, which lives in %SYSTEMROOT%\\framework\version\config. The key settings are in the processModel block, where there are parameters to control the number of IO and worker threads. The relevant parameter names are: "enable", "maxWorkerThreads" and "maxIOThreads". Changing the enable parameter to false means that ASP.NET will run in the same application process as IIS, rather than in its own process.

In Windows Server 2003, the enable parameter is ignored and the IIS 6 process model is always used. With IIS 6, each Web site is assigned to an application pool, and each pool runs in its own process space. Figures 5 and 6 show how the IIS 6 application pools can be configured.


Figure 5: Configuring Windows Server 2003 as an application server


Figure 6: Configuring the number of worker processes

Table 2 details all the parameters that can be used to tune ASP.NET Web applications, including Web services.

Table 2. Tuning parameters

Tuning ParameterDescriptionValue Used
Web.config: SessionStateIndicates the type of store to use for session information, or whether to turn off session state altogether. Options are Off, InProc, StateServer, or SQLServer.Off
web.config: EnableSessionStateIndicates the default as to whether the pages in the Web application will access session state.

The equivalent in JSP pages would be:

<%@ page session="false" %>

web.config: EnableViewStateIndicates the default as to whether the pages should track view state information between page loadings.False

compilation debug

Indicates the default as to whether the pages should be compiled in debug or release mode.

Most servlet engines in J2EE will allow you to specify compile options for JSP compilation.

ADO.NET Connection string Min Pool Size and Max Pool SizeConfigures the size of the internal ADO.NET connection pool.

In J2EE, most database connections are accessed through a JNDI DataSource, and most application servers allow for the configuration of the size of the connection pool used for the DataSource.

Typically min and max were set to the same value of 5.
machine.config processModel enable

(Windows 2000 only)

Indicates whether ASP.NET should run in its own process or in the inetinfo.exe process.

In Java, most J2EE application servers can be fronted by the Apache Web server, IIS or iPlanet etc. while others can be accessed via the internal http listener that runs in process with the JVM.

Machine.config processModel maxWorkerThreadsMaximum number of worker threads per CPU in the thread pool.

In J2EE, most servlet containers allow some form of configuration of the thread pool size.

Dependent on number of CPUs in the system, but typically we reduced this below the default of 20.
Machine.config processModel maxIOThreadsMaximum number of IO threads per CPU in the thread pool.

In J2EE, most servlet containers allow some form of configuration of the thread pool size.

Dependent on number of CPUs in the system, but typically we reduced this below the default of 20.
Application Pool

(Windows Server 2003 only)

A container for Web sites within IIS. Each application pool is process-isolated from other application pools, and has its own process monitoring system.Dedicated application pool created for application.
Application Pool:

Recycle work process

(Windows Server 2003 only)

Indicates how often the worker process should automatically be recycled.Off
Application Pool:

Idle Time

(Windows Server 2003 only)

How long the application pool should run if no requests are made to one of the contained Web sites.Off
Application Pool:

Request Queue limit

(Windows Server 2003 only)

How many pending requests the pool will allow before rejecting the request and sending an HTTP 503 response.Off
Application Pool: Web Garden worker processes

(Windows Server 2003 only)

How many worker process the application pool should use to process requests. By running more than one process, IIS can provide a more robust service. For example, the second process will keep running if the first stalls while processing a script. Contention for resources is reduced as each pool as its own set of resources.

In Java, J2EE application servers can employ JVM cloning to provide fault tolerance and improved performance by reducing resource contention.

Application Pool: Enable pinging

(Windows Server 2003 only)

Indicates how often IIS should ping the application process to check the status of the process.Off
Application Pool:

Enable Rapid Fail Protection

(Windows Server 2003 only)

Indicates if IIS should disable a process if a certain number of application failures occur in a specified time interval.Off


Building Web Services with JBoss

JBoss Application Server version 3.2 comes with the jboss-net module in the "all" configuration for deploying Java Web services. Jboss-net is a packaged deployment of the Apache Axis project, and supports Java Web services features such as jws pages and Web service descriptor files (Web services equivalent to the web.xml file for Web applications, or the ejb.jar file for EJBs).

The application design for the Java implementation of the application followed the same pattern as the .NET version. The application was broken into 3 tiers; a Web services façade tier which contained a java class used to wrap access to the business logic classes, a business logic tier implemented as stateless session EJBs and a set of data access layer classes which encapsulated all the JDBC calls.


Figure 7: Java/J2EE application architecture

Figure 7 shows the application structure of the Java implementation. For the JBoss version we created a Java class for the Web service façade, which was described using the web-services.xml listed below. JWS pages do not support the serialization of custom bean classes and so the bean mappings needed to be specified using the XML configuration file.

<!-- Doculabs Web Service Descriptor -->

   <service name="wsFacade" provider="java:RPC">
      <parameter name="className" 
      <parameter name="allowedMethods" 
                value="getCustomer insertCustomer getOrderDetails 
                       getCustomerPlain insertCustomerPlain

      <beanMapping qname="tns:Customer"   

      <beanMapping qname="tns:Address" 

      <beanMapping qname="tns:Order" 

      <beanMapping qname="tns:LineItem"


Some concern was raised that using EJBs might unfairly hinder the performance of the Java implementations. To mitigate this we developed two middle-tier versions of the Java applications. The first version used EJBs to contain the business logic, and in the second version we used plain old Java beans. Due to the simplicity of the test there were very few code differences between the EJB version and the non-EJB version (plain version) apart from the required interface files and inherited objects required for the EJB version.

Consuming a JBoss (jboss-net) Web Service in .NET

The following section steps through the process of consuming the jboss-net Web service in an ASP.NET Web application built in Visual Studio .NET 2003.


Figure 8: Adding Web references to a Visual Studio .NET 2003 project

Figure 8 shows how to start the process of adding a Web service "Web Reference" to an ASP.NET project.


Figure 9: Importing the JBoss-net WSDL file

Launching the wizard brings up the dialog box, shown in figure 9, which allows you to enter the location of a known WSDL file or helps you search for a UDDI server.

When you click the Add Reference button, Visual Studio .NET will import a snapshot of the WSDL file into your project and generate a proxy class that will enable you to invoke the Web service from your client application.


Figure 10: Marking the Web reference as Dynamic

To help avoid hard-coded references to the Web service in your application code, Visual Studio .NET enables you to mark a Web service as Dynamic, as shown in Figure 10. By setting the Web service to Dynamic, a new parameter is created (in web.config or the relevant configuration file), which can be used to update the location of the Web service. This means that when migrating the application from dev to test, and test to production, all that is required is for an administrator to update the configuration file to point to the appropriate Web service instance.

   <add key="DoculabsClients.JBoss-Server.wsFacadeService" 

Once the Web service reference has been added in Visual Studio .NET, you can use the Web service proxy classes in your code. As with all classes in Visual Studio .NET, you will get full IntelliSense on the Web Service Proxy classes and related data classes, making you more productive with the new Web service.

JBoss_Server.wsFacadeService webService 
   = new JBoss_Server.wsFacadeService();

JBoss_Server.Order order = webService.getOrderDetails(1);

// Do something with the order
foreach(JBoss_Server.LineItem line in order.lineItems)
   //Do something for each line item in the order

If you prefer, you can use the command-line tool wsdl.exe to generate the Web service proxy classes. Wsdl.exe is located in the bin directory of the Framework SDK directory.

C:\Code\wsdltest>wsdl http://localhost:8080/jboss-net/services/wsFacade?wsdl /out:wsFacadeServiceProxy.cs
Microsoft (R) Web Services Description Language Utility
[Microsoft (R) .NET Framework, Version 1.1.4322.573]
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

Writing file 'C:\Code\wsdltest\wsFacadeServiceProxy.cs'.


Consuming a .NET Web Service in JBoss

To use the .NET Web service from within a JBoss application you need to run a utility similar to the wsdl utility mentioned above. The Axis Java equivalent is WSDL2Java, which is contained in the axis.jar file that is part of the Axis distribution. It is bundled in jboss-net.sar of the JBoss "All" configuration.

C:\JavaDev\axisClient>java org.apache.axis.wsdl.WSDL2Java http://localhost/WebService/wsFacade.asmx?WSDL

For the .NET Web service WSDL file, the WSDL2Java command creates 9 files as listed in Table 3.

Table 3. WSDL2Java command files

Address.javaThin data class used to serialize the address object
ArrayOfLineItem.javaThin data class used to serialize arrays of line items
Customer.javaThin data class used to serialize the customer object
LineItem.javaThin data class used to serialize the line item object
Order.javaThin data class used to serialize the order object
WsFacade.javaThe Service Interface, which contains methods to access the Web service proxy
WsFacadeLocator.javaThe Locator Service implements the Service Interface and is responsible for instantiating the service stubs.
WsFacadeSoap.javaA Service Definition Interface. This is the class that should be used to access methods on the Web service.
WsFacadeSoapStub.javaThe stub class, which implements the Service Definition Interface. It acts as a proxy for the remote Web service so that users can treat the Web Service Objects as local objects.

These classes can be compiled into a jar file and deployed with the JBoss client application. To use the Web service proxy, you first need to create an instance of the locator. Using the locator class, you can create an instance of the SOAP service, with the option of passing in a different Web service endpoint. The code snippet below shows the creation of the locator, the retrieval of the SOAP service, and calling the getOrderDetails method.

//Create an instance of the locator class
WsFacade service = new WsFacadeLocator(); 

//Set up a URL to hold the required Web service endpoint
URL url = new URL("http://localhost/WebService/wsFacade.asmx")

//Get the SOAP Service at the specified endpoint
WsFacadeSoap ws = service.getwsFacadeSoap(url); 

//Call the GetOrderDetails Web service method
Order customer = ws.getOrderDetails(1);


The .NET Web service and Web service client were tuned in exactly the same way using a combination web.config, machine.config and IIS process configuration to ensure that all the CPUs on the server were fully utilized.

Tuning the Java server required experimentation with the following parameters.

Table 4. Parameters

ParameterDescriptionValues Used
JVMSeveral parameters are available when tuning the JVM: vendor and version, heap size, garbage collection mode, number of JVM instances, and processor affinity of JVMs.JVM version: 1.4.1

Single JVM instance

1.5GB Heap size min=max

Parallel garbage collector

Servlet ContainerJBoss supports both Jetty and Tomcat as the servlet container.Jetty, thread pool size set to 700

Results Summary

The key results from the three tests are presented below. For the complete set of test results, please visit the Doculabs Web site.

Direct to Web Service


Figure 11: Performance of Web service host

Figure 11 shows the results when Mercury Load Runner is used to make SOAP calls directly to the Web service host.

Web service host machine: HP 8-way 900 MHz, DL 760 with 4GB of RAM

Web Service Client Stress Test


Figure 12: Performance of Web service proxy

Figure 12 shows the results when Mercury Load Runner is used to make Web requests to a simple Web application running on one machine. The Web application machine is also running a Web service proxy to the Web service used in the "Direct to Web service test". Each page request in the Web application requires a Web service call to be made from the first machine to the second machine.

Web application host, Web service proxy machine: HP 4-way 1.4 GHz, DL 580 with 2GB RAM

Web service host machine: HP 8-way 900 MHz, DL 760 with 4GB of RAM


Figure 13: Comparison of interoperability results with single-vendor solution

Figure 13 shows the results of the previous test compared with the results of the interoperability runs (orange bars), where a .NET Web service is consumed by a Java Web service proxy, and a Java Web service is consumed by a .NET Web service proxy.

Web application host, Web service proxy machine: HP 4-way 1.4 GHz, DL 580 with 2GB RAM

Web service host machine: HP 8-way 900 MHz, DL 760 with 4GB of RAM


The results demonstrate that Microsoft .NET Web services, Windows Server 2003 and Microsoft Visual Studio .NET provide an excellent platform for developing and deploying Web services. They also show that Microsoft .NET performs very well in environments where customers have chosen to implement Oracle as the back-end data store or need to leverage previous investments in Oracle.