Stress Testing

Custom LoadGenerator Tool Identifies the Issues Your Application Faces Under Stress

Brian Otto

Code download available at:StressTestingTool.exe(164 KB)

Contents

Getting Started
Creating a Worker
Inside the LoadGenerator App
Initializing a Simulation
Running the Simulation
Completing the Simulation
Conclusion

In order to create a reliable application you must understand how various components will perform under heavy loads and ensure that the system behaves appropriately. Server components for popular Web sites need to respond to millions of requests per day. Corporate intranets and desktop applications may only need to handle a few hundred or a few thousand requests per day. Many smaller development organizations feel they don't have the time or money to devote resources to building out environments to conduct rigorous stress tests, even though much of the success of an application can be attributed to how consistently it responds to users. If you don't take the time to understand how your application will respond to extreme use and failures, it could lead to your application's demise.

Stress testing should be conducted at all stages of development, including the stage before individual components are incorporated into a larger application to prevent issues later in the development cycle. Having the right tools available to perform this initial set of tests can make the task quick and easy.

Getting Started

The Microsoft® .NET Framework provides a great foundation to build a reusable and extensible stress tool that makes it easier to test your application. The application in this article uses reflection to dynamically load a custom class in an assembly that you create. The class performs a set of tasks representing a desired test. You control how many instances of the class are loaded and how many times the application calls into each instance during the simulation. Each instance of the class is instantiated on a separate thread. Since the actual tasks being performed are in a custom assembly that you create, you have the flexibility to create worker components that stress just about any application layer, be it a Web service, a database, or a COM+ serviced component. This stress-testing tool can help you verify that your application meets both its performance and concurrency requirements, as well as help you determine how it responds to stressful loads.

Figure 1** Load Generator **

Figure 1 shows the main form of the application. In the Test Setup frame, you can configure how the simulation will be conducted. The Workers setting identifies how many instances (threads) will be created for executing the test. The number of iterations for each instance is set through the Cmds/Worker value. Think time represents the delay between successive iterations for each instance. When conducting a stress test, it is also important to control the warm-up time. Most systems have required initialization that must be performed before any work can be done (for example, ASP.NET pages are compiled the first time they're accessed). During the warm-up period, no performance data is collected so as to minimize the distortion of the test results. The information in the Statistics frame provides a means to monitor the simulation. When the test is complete, a status form appears showing some statistics for each instance (see Figure 2).

Figure 2** Performance Stats **

If you checked the Show Detailed Progress box during the simulation on the main form, this status form will be visible during the test. Because updating this form during the test will consume resources, it should be done sparingly when simulating a heavy load.

Creating a Worker

To help maintain better control over the execution of the simulation, the application does not use the thread pool when spawning worker threads. Each thread is created and managed manually. If I had used the thread pool, new requests might have been queued if CPU activity was high or no free threads existed in the thread pool. Since this is a stress-testing tool, I wanted to maintain direct control over how much activity is generated based on the test setup options.

Creating a worker thread involves creating a class in an assembly that implements the LoadGenerator.Interfaces.IWorker interface, as shown here:

public interface IWorker: IDisposable { void Initialize(long instanceID); void Process(); }

As you can see from the definition, your class needs to implement the IDisposable interface. The Dispose method will be called at the end of the simulation to provide your class an opportunity to perform cleanup tasks. The Initialize method is called only once, before any tests are executed. The instanceID parameter represents the unique instance number of the worker relative to all the workers being created. The Process method is where the code that performs the work for the simulation should reside. This method will be called repeatedly as defined by the Test Setup settings on the main form.

For this example, I will create a worker component that stresses a stored procedure in the Northwind database. SQL Server™ and Northwind are assumed to be on the same machine as the LoadGenerator application. In a real test, however, SQL Server and Northwind would reside on a separate machine. On a single CPU system, running the client load generator and SQL Server on the same machine can quickly overtax the computer and defeat the intent of the test. In addition, it is perfectly viable to have a test client running on multiple machines.

Start by creating a new C# library assembly called DBWorker in Visual Studio® .NET. Add a reference to the LoadGenerator.Interfaces assembly and change the default class declaration to the following line of code:

public class MyWorker : LoadGenerator.Interfaces.IWorker

At the end of the line, Visual Studio should prompt you to press the Tab key, which will cause the stubs for all the required methods of the IWorker interface to be created for you automatically.

Figure 3 shows the Initialize, Process, and Dispose method implementations. A class-level Command object is created in the Initialize method and reused across calls to the Process method. Reducing the unnecessary overhead within each execution of the Process method helps improve the efficiency and effectiveness of the test. The Initialize method configures the Command object to call the [employee sales by country] procedure in the Northwind database with two dates as arguments. The Process method opens a pooled database connection, executes the command, and releases the connection back into the pool. A more realistic test would probably have the Process method change the arguments to the procedure on subsequent executions.

Figure 3 DBWorker Class Implementing IWorker

public void Initialize(long lInstance) { //Initialize a command object _cmd = new SqlCommand(); _cmd.CommandText="[employee sales by country]"; _cmd.CommandType = CommandType.StoredProcedure; _cmd.Connection = new SqlConnection("Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=Northwind;Data Source=(local)"); _cmd.Parameters.Add("@Beginning_Date", "1/1/1997"); _cmd.Parameters.Add("@Ending_Date", "1/1/1998"); } public void Process() { try { _cmd.Connection.Open(); _cmd.ExecuteNonQuery(); } finally { _cmd.Connection.Close(); } } public void Dispose() { if (_cmd != null) { _cmd.Connection.Close(); _cmd.Dispose(); _cmd = null; } }

The stored procedure is executed without regard for the resultset the procedure returns. If you wanted to test the performance impact of loading the resultset into a DataTable, you could compare this test to one that filled a DataTable. Just be aware of how much client work is being performed for each test since it may consume CPU time and limit the client's ability to execute additional tests.

Now you can compile the assembly and copy it to the application directory. The application directory contains an application configuration file that tells the application what assembly and class to load when conducting a test. Open the LoadGenerator.exe.config file and add the name of your assembly and class:

<configuration> <configSections> <section name="Worker" type="System.Configuration.SingleTagSectionHandler"/> </configSections> <Worker assembly="DBWorker" type="DBWorker.MyWorker"/> </configuration>

After updating the configuration file, you are ready to run the simulation. Start up the LoadGenerator application and change the number of workers to one so you can verify that everything is working; then click the Start button. If there are any errors when activating your component or executing the tests, they will be displayed in the Status field of the status form. If everything worked, you are ready to run more instances and change the think time and warm-up periods to vary the stress on the system. That's all there is to making this tool work. To get the most out of this example, the Northwind database should reside on a different computer. More sophisticated worker components can perform a variety of tasks to stress your applications.

Inside the LoadGenerator App

The application divides most of the simulation work into two classes. The WorkerState class houses the code that creates an individual thread and provides access to start and stop its execution as well as to report state information. This class also loads the custom class and calls to the IWorker interface methods. For every instance of a worker defined in the test setup, one instance of the WorkerState class is created. The WorkerCollection class is the container for all of the WorkerState objects created for a given simulation. This class controls the starting and stopping of the simulation and exposes one event called TestComplete that provides notification to the forms when a simulation has completed.

To conduct a simulation, the application first initializes the WorkerCollection object and then calls WorkerCollection.Start. Much of the initialization is performed asynchronously from the main application thread from which the WorkerCollection is accessed. Because of this, the Start method blocks until initialization is complete to ensure that no tests are conducted while some threads are still initializing. The initialization of the WorkerCollection object includes the creation and initialization of WorkerState objects for each worker defined by the simulation. Initialization of each WorkerState object creates a new thread and loads the custom assembly. Once all threads have loaded the custom assembly, the WorkerCollection.Start method stops blocking and each thread is allowed to proceed to the warm-up cycle and begin making calls into the custom assembly to conduct tests.

The following sections step through the initialization, running, and completion tasks.

Initializing a Simulation

When a simulation is started by clicking the Start button on the main form, the form hooks up to the TestComplete event of the WorkerCollection object and calls the WorkerCollection.Initialize and WorkerCollection.Start methods. As you will see a little later, the Start method may take up to 30 seconds to return. To keep the application more responsive to the user, the Start method is called asynchronously:

_workerCollection = new WorkerCollection(); _workerCollection.TestComplete +=new TestCompleteEventHandler(_workerCollection_TestComplete); _workerCollection.Initialize(Convert.ToInt32(txtInstances.Text), Convert.ToInt32(txtWarmup.Text), Convert.ToInt32(txtInterval.Text), Convert.ToInt64(txtMaxCmds.Text)); StartDelegate _startSimulation = new StartDelegate (_workerCollection.Start); StartDelegate.BeginInvoke(null,null);

The WorkerCollection.Initialize method takes as arguments all the test setup parameters specified on the main form:

void Initialize(int totalWorkers, int warmUpTime, int thinkTime, long maxCommands)

This method then looks into the application configuration file to retrieve the class that will be doing all the actual stress tests and passes it on to each of the WorkerState objects. The WorkerState objects are created to control execution of each thread and retrieve state information (see Figure 4).

Figure 4 Get State Information

WorkerState workerState = new WorkerState(); workerState.InstanceID = index; workerState.IWorkerAssemblyName = assemblyName; workerState.IWorkerTypeName=typeName; workerState.ThinkTime = thinkTime; workerState.MaxCommands = maxCommands; workerState.WarmUpDelay = (int)(((double)_warmUpTime/_totalCount) * (double)index * 1000); workerState.Initialize(_syncStart, new WorkerCallBack(WorkerInitialized), new WorkerCallBack(WorkerStarted), new WorkerCallBack(WorkerComplete)); _workerCollection[index] = workerState;

After setting the running parameters on the WorkerState object, the WorkerState.Initialize method is called. The thread created by this method will wait to be signaled by the application before conducting any stress tests. The _syncStart argument is a ManualResetEvent that will be signaled after all WorkerState object instances have completed their initialization tasks. The remaining arguments to the Initialize method are callback delegates from the WorkerState object to the WorkerCollection object. The calling of these delegates will indicate to the WorkerCollection when each worker has completed initialization, started conducting tests, and finished conducting tests. As you'll see, the WorkerCollection object uses these callbacks to track the progress of the simulation.

The WorkerState.Initialize method creates and starts the worker thread, specifying which method the thread should execute:

ThreadStart ts = new ThreadStart(ProcessCmd); _thread = new Thread(ts); SetActionState(ActionEnum.Initialize); _syncStart = startWaitHandle; _thread.Start();

The actions of the worker threads are controlled through the _actionState variable, whose type is an enumeration called ActionEnum. This enumeration helps identify the task the thread is performing and what task the thread is supposed to perform. If the simulation is stopped by clicking the Stop button on the main form, the WorkerCollection object will set the variable to ActionEnum.Stop. As the thread proceeds though its tasks, it regularly checks the value of this variable. If the enumerated value is Stop, the thread will exit gracefully. Since this value is updated by both the worker thread and the main application thread, all updates to this variable are synchronized and performed by one method called SetActionState.

Once Thread.Start is called, the WorkerState.ProcessCmd method begins executing. This method creates an instance of the custom class specified in the configuration file. The ProcessCmd method then indicates its completion of initialization tasks by issuing a callback to the WorkerCollection and waiting on a signal from the _syncStart ManualResetEvent:

Assembly workerAssembly = Assembly.Load(_IWorkerAssemblyName); objWork = (IWorker)workerAssembly.CreateInstance(_IWorkerTypeName, true); _initCallBack(_instanceID); _syncStart.WaitOne();

The method invoked in the WorkerCollection object by the _initCallBack delegate will signal the _syncInitialized wait handle when all workers have completed their initialization work:

lock(this) { if (++_initCount == _workerCollection.Length) _syncInitialized.Set(); }

This completes the initialization phase of the simulation. At this point, each thread has created an instance of the worker component and is now waiting on _syncStart.WaitOne. The signaling of this wait handle, which happens in the WorkerCollection.Start method, will start the simulation.

Running the Simulation

The WorkerCollection.Start method is called from the main form after the WorkerCollection.Initialize method returns. Since many of the initialization steps are performed by the individual worker threads asynchronous to the Initialize method, the Start method waits until all threads have finished their initialization tasks. This method waits up to 30 seconds on the _syncInitialized wait handle. As you saw earlier, this wait handle is signaled after the last thread has invoked the _initCallBack delegate. If the WorkerCollection times out while waiting for all the workers to initialize, the simulation is stopped and all threads will exit, as you can see in the code listed in Figure 5.

Figure 5 Waiting

public bool Start() { try { if (_syncInitialized.WaitOne(30000,false) == false) throw new Exception("Timeout waiting for initialization."); _syncStart.Set(); return true; } catch (Exception err) { Stop(); return false; } }

If _syncInitialized is signaled before the timeout, the Start method will in turn signal the _syncStart ManualResetEvent. This is the wait handle that each of the worker threads are waiting on. Once signaled, the worker threads proceed beyond the Initialized state and begin the warm-up cycle:

_syncStart.WaitOne(); if (_actionState == ActionEnum.Stop) return; DelayForWarmUp(); objWork.Initialize(_instanceID); _startCallBack(_instanceID); SetActionState(ActionEnum.Process);

After being signaled to start the warm-up delay, the worker thread first checks to verify that the simulation wasn't stopped and then proceeds to sleep the appropriate amount of time to accommodate the warm-up period. Each thread has a different warm-up period (delay) that it needs to spend sleeping. These delays are assigned to ensure that threads begin stress tests at regular intervals with all threads active at the end of the warm-up period. As I mentioned, no metrics on performance are captured during this warm-up time. If the warm-up period is 10 seconds and there are 10 threads, 1 thread will begin a test every second. The first thread will have a _warmUpDelay of 0, and the tenth thread will have a _warmUpDelay of 9 seconds. The threads do not stay asleep for the entire warm-up delay period. The DelayForWarmUp method runs in a loop, sleeping for short intervals of no more than 500 milliseconds until the warm-up time has elapsed:

private void DelayForWarmUp() { DateTime startTime = DateTime.Now; TimeSpan totalDelay = TimeSpan.Zero; int remainingTime = _warmUpDelay; while(remainingTime > 0 && _actionState != ActionEnum.Stop){ Thread.Sleep(remainingTime > 500 ? 500 : remainingTime); totalDelay = DateTime.Now.Subtract(startTime); remainingTime = (int)(_warmUpDelay - totalDelay.TotalMilliseconds); } }

Sleeping in this manner allows the thread to stay responsive to a stop request from the user without having to resort to the use of the Thread.Interrupt method. The Thread.Interrupt method injects an exception into the sleeping thread and wakes it from its sleep. This approach is a bit costly when the simulation has a few hundred threads. If the _actionState variable is updated to Stop, DelayForWarmUp exits the loop and returns to ProcessCmd, which will allow the thread to exit gracefully.

After the warm-up delay is over, the IWorker.Initialize method is called to allow the worker class to prepare for the tests. The _startCallBack delegate is then invoked, informing the WorkerCollection that the thread is beginning its tests. The _actionState variable is updated accordingly.

In the WorkerCollection object, the callback method invoked by the _startCallBack delegate only needs to track the number of threads that have moved beyond their respective warm-up delay and have begun running stress tests. The Interlocked class in the System.Threading namespace provides a very simple and efficient mechanism to synchronize variable incrementing. The method calls Interlocked.Increment(ref _activeCount) to increment the number of active threads. This value will be queried by the forms to show progress.

With the custom class initialized, the worker thread proceeds with the execution of the stress tests:

while(_iterations < _maxCmds && _actionState == ActionEnum.Process){ _iterations++; objWork.Process(); if(_thinkTime > 0) Thread.Sleep(_thinkTime); }

As long as the current action state is Process and there are more iterations left to go, the IWorker.Process method is called, and the thread sleeps for the required period of think time between each call. When the loop exits, the code in the finally block executes to clean up the instances, as shown here:

finally { _completeCallBack(_instanceID); if (objWork != null) objWork.Dispose(); objWork = null; }

This invokes a delegate that calls back into the WorkerCollection object to notify completion of the tests. IWorker.Dispose is then called to allow the custom component to perform any clean-up tasks before being released.

Completing the Simulation

When the last thread has called the _completeCallBack delegate, the TestComplete event is invoked in the WorkerCollection object (see Figure 6). The TestComplete event is invoked on whichever thread happens to be the last to complete. The forms are hooked into this event to allow them to display any final information and reset themselves for the next simulation. For the main form and status form, the completion code needs to be invoked on the thread that owns the form:

private void _workerCollection_TestComplete(object sender, TestCompleteEventArgs e) { testCompleteEventHandler d = new testCompleteEventHandler(this.TestComplete); this.Invoke(d,new object[] {e}); }

Figure 6 Is Test Complete?

bool bComplete=false; WorkerState workerState = (WorkerState) sender; lock(this) { if (workerState.ThreadError) _errorCount++; if (_activeCount > 0) _activeCount—; if (++_completeCount == _totalCount) bComplete = true; } if (bComplete) { TestCompleteEventArgs args = new TestCompleteEventArgs("", _duration.TotalSeconds); TestComplete(this, args); }

The form event handlers for the WorkerCollection.TestComplete event create a new delegate that points to a private method in the TestComplete form class, which is then invoked on the same thread that controls the form. The main form resets itself for a new test and the status form is displayed with the individual running times and number of iterations for each thread.

Conclusion

This article would not be complete without a mention of the importance of security when loading late-bound assemblies. Although you won't likely find third-party assemblies that implement my IWorker interface, it is important to understand how to load late-bound assemblies in a secure manner. Jason Clark has a great article in the October 2003 issue of MSDN® Magazine covering this topic in the context of writing plug-in frameworks for your .NET-based applications. If you plan to use this functionality in your applications, I highly recommend you read his article.

The LoadGenerator application simplifies the tasks of conducting basic stress tests by providing the core plumbing needed to simulate high concurrency. The components you create that implement the IWorker interface need not worry about this plumbing, allowing you to focus on understanding the behavior of your application under stress. Your components can test any number of scenarios, including the efficiency of a new algorithm, the throughput of a Web service, or the concurrency of a database procedure. What you make the component do is up to you.

Brian Otto is a Senior Development Consultant at Microsoft, specializing in .NET and SQL Server development. He has created a variety of software solutions in the financial, automotive, and media industries.