TN_1301: Implementing Sync Points in Visual Stu...

TN_1301: Implementing Sync Points in Visual Studio Team System Load Tests

Bill Barnett, Software Design Engineer
Microsoft Corporation

Visual Studio Team System load tests do not have a "sync point" feature that lets a load tester specify that some number of test instances for different virtual users should all sync up at the same point of execution in the test.

However, with some caveats it is possible to implement sync points in a unit test or Web test that is run within a Visual Studio Team System load test.   This article includes sample code for doing this in both a unit test and a Web test.  Both of these samples use a common SyncPoint class which is also included below.

I mentioned caveats; perhaps the biggest caveat, which applies whether the load test contains Web tests or unit tests is that the sample implementations only sync up the execution of tests running on the same load test agent; it does not synchronize tests running on different agents.  There are other caveats specific to Web tests which are described below.

The SyncPoint class

// The SyncPoint class used to implement sync points in both the unit test and Web test examples.
// Add this to a C# test project (you may want to change the namespace)
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
namespace TestProject1
{
    class SyncPoint
    {
        private object m_syncPointLock = new object();
        private int m_syncPointUserCount;
        private int m_waiterCount;
        private ManualResetEvent m_syncPointEvent = new ManualResetEvent(false);
        // Constructor
        // The syncPointUserCount argument specifies the number of calling threads
        // that must call the Wait() method before the Wait() method will return
        // (except in the case of a timeout)
        public SyncPoint(int syncPointUserCount)
        {
            m_syncPointUserCount = syncPointUserCount;
        }
        // Wait for syncPointUserCount number of threads to reach this point,
        // but timeout after the specified number of milliseconds
        public void Wait(int timeoutInMilliseconds, bool throwExceptionOnTimeout)
        {
            Monitor.Enter(m_syncPointLock);
            if (++m_waiterCount < m_syncPointUserCount)
            {
                Monitor.Exit(m_syncPointLock);
                if (!m_syncPointEvent.WaitOne(timeoutInMilliseconds, false) && throwExceptionOnTimeout)
                {
                    m_syncPointEvent.Set();
                    throw new ApplicationException("Timed out waiting for sync point");
                }
            }
            else
            {
                m_syncPointEvent.Set();
                m_waiterCount = 0;
                m_syncPointEvent = new ManualResetEvent(false);
                Monitor.Exit(m_syncPointLock);
            }
        }
    }
}

Configuring a Load Test for the sync point examples

Both the unit test and Web test examples that follow assume that the number of users to sync up is specified in the load test as a LoadTest Context Parameter named "SyncPointUserCount".

To add the context parameter in the Load Test Editor, follow these steps:

  1. Right-click the Active Run Settings node in the tree

  2. Select Add Context Parameter - you will see the property sheet

  3. For the parameter name, type SyncPointUserCount

  4. For the parameter value, specify the number of virtual users to sync up

Make sure that the value for SyncPointUserCount is not larger than the user load for the Scenario that contains the test that uses the sync point.   If the value for SyncPointUserCount is larger, then when the tests reach the sync point, they will all hang until the sync point timeout is reached.  This occurs because there will never be enough threads that reach the sync point.

Load Test Plug-in

For the SyncPointUserCount parameter to be passed down to the unit tests or Web tests contained in the load test, add the following Load Test Plug-in class to your test project. Next, set this as the Load Test plug-in for the Load Test that is to contain the sync point.

If you already have a Load Test Plug-in defined, just add the code from the TestStarting event handler that passes the Load Test context parameters to the contained tests.

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using Microsoft.VisualStudio.TestTools.LoadTesting;
namespace TestProject1
{
    /*
      This class is a sample LoadTest Plug-in class that copies any Context Parameters defined in the
      Load Test into the test context for any unit test or Web test contained in the load test.
      To use this class:
      1. Add the class to a test project.
      2. Build the test project.
      3. Hook this load test plugin up to your load test:
         In the load test editor, select the top-most node in the load test tree.
         This is the load test name in the tree.
         Go to the property sheet.
         Click on the symbol next to the value for "Load Test Plug-in".
         Select the ContextParametersLoadTestPlugin class as your plugin.
     */
    public class ContextParametersLoadTestPlugin : ILoadTestPlugin
    {
        public ContextParametersLoadTestPlugin()
        {
        }
        public void Initialize(LoadTest loadTest)
        {
            m_loadTest = loadTest;
            m_loadTest.TestStarting += new EventHandler<TestStartingEventArgs>(TestStarting);
        }
        private void TestStarting(object source, TestStartingEventArgs testStartingEventArgs)
        {
            foreach (KeyValuePair<string, object> keyValuePair in m_loadTest.Context)
            {
                testStartingEventArgs.TestContextProperties.Add(keyValuePair.Key, keyValuePair.Value);
            }
        }
        private LoadTest m_loadTest;
    }
}

Unit Test Example

// This is a very simple example of a unit test that uses a sync point so that when running the unit test in a load test
// a specified number of virtual users running the unit tests will all sync up at the same point in the test code.
// Note that waiting for the sync point will only occur if running the unit test in a load test that defines the
// SyncPointUserCount context paramter.
// This allows you to run the unit test as a standalone test without it waiting.
using System;
using System.Text;
using System.Threading;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace TestProject1
{
    [TestClass]
    public class UnitTest2
    {
        private static SyncPoint s_syncPoint = new SyncPoint(1);
        private static Random s_random = new Random();
        public UnitTest2()
        {
        }
        // Use ClassInitialize to run code before running the first test in the class
        [ClassInitialize()]
        public static void MyClassInitialize(TestContext testContext)
        {
            if (testContext.Properties.Contains("SyncPointUserCount"))
            {
                string sSyncPointUserCount = (string)testContext.Properties["SyncPointUserCount"];
                try
                {
                    s_syncPoint = new SyncPoint(int.Parse(sSyncPointUserCount));
                }
                catch (FormatException)
                {
                    throw new ApplicationException(
                        "The value specified for SyncPointUserCount is not in integer format: " + sSyncPointUserCount);
                }
            }
        }
        [TestMethod]
        public void SyncPointExampleTestMethod()
        {
            // Sleep a random time between 0 and 5 seconds
            Thread.Sleep(s_random.Next(5000));
            // Wait for 50 tests to reach this point (timeout after 30 seconds)
            s_syncPoint.Wait(30000, true);
        }
    }
}

Web Test Example

The following example is a C# coded Web test that uses a sync point to sync up a specified number of Web test virtual users at a particular request in the Web test.

However, there are some limitations on using sync points with Web tests that arise from the thread model that the load test runtime engine uses when it runs Web tests.

To make a long story short, an important optimization in the load test runtime engine is that it uses async I/O to send Web requests.  This allows the load test engine to use thread pooling to multiplex a large number of virtual users onto a smaller number of threads.  This, in turn, reduces the amount of memory required by the load test runtime engine and improves the efficiency.

The drawback to this thread model is that user-written Web test code should generally not perform any processing that will pend the thread, because the thread is likely to be a thread pool thread that is needed to process requests for other virtual users.

There is one partial exception to this rule: the post request event handler may perform operations that pend as long as number of simultaneous threads pended is not more than 50 or so.  This is because the post request event handler is called on a thread from the System.Net I/O completion thread pool which can have a fair number of threads.

All of this implies two restrictions on the use of sync points in Web tests:

  1. The call to wait on the sync point must be in the post request event handler.   This means, for example, to sync up the Web tests before issuing the 5th request in your Web test, you actually must define the sync point to occur after the completion of the 4th request.   Note that if there is think time between these requests and the Load Test's think profile is set to use the "Normal Distribution" option, the variance in think times for different users may result in the subsequent requests being somewhat out of sync.   To avoid this, the think profile could be changed from "Normal Distribution" to "On."

  2. The number of virtual users to sync up (as specified by the "SyncPointUserCount" load test context parameter described earlier) should not be more than about 50.   Specifying a greater value might cause the load test to hang.  (It might be possible to use a greater value if you are running the load test on a multi-processor computer.)

Also, I would stronly suggest that you set the Connection Model in the load test's Run Settings to Connection Per User so that the requests that you want to sync up are actually all submitted in parallel (some requests might get queued behind others if a Connection Pool is used).

namespace TestProject1
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Threading;
    using Microsoft.VisualStudio.TestTools.WebTesting;
    using Microsoft.VisualStudio.TestTools.WebTesting.Rules;
    public class SyncPointSampleCoded : WebTest
    {
        // Default the value of s_syncPointUserCount to 1 so that it runs without waiting when run
        // as a standalone Web test
        private static SyncPoint s_syncPoint = new SyncPoint(1);
        private static bool s_syncPointUserCountSet;
        public SyncPointSampleCoded()
        {
            this.PreAuthenticate = false;
            this.PreWebTest += new EventHandler<PreWebTestEventArgs>(SyncPointSampleCoded_PreWebTest);
        }
        void SyncPointSampleCoded_PreWebTest(object sender, PreWebTestEventArgs e)
        {
            // If the Context property SyncPointUserCount is found, use this value
            // This would normally be set by defining this as a Load Test context parameter
            if (!s_syncPointUserCountSet && this.Context.ContainsKey("SyncPointUserCount"))
            {
                s_syncPointUserCountSet = true;
                string sSyncPointUserCount = (string)this.Context["SyncPointUserCount"];
                try
                {
                    s_syncPoint = new SyncPoint(int.Parse(sSyncPointUserCount));
                }
                catch (FormatException)
                {
                    throw new ApplicationException(
                        "The value specified for SyncPointUserCount is not in integer format: " + sSyncPointUserCount);
                }
            }
        }
        public override IEnumerator<WebTestRequest> GetRequestEnumerator()
        {
            WebTestRequest request1 = new WebTestRequest("http://vsncts02/StoreCSVS/Static/default.htm");
            request1.ThinkTime = 2;
            yield return request1;
            WebTestRequest request2 = new WebTestRequest("http://vsncts02/StoreCSVS/Static/productslist1.htm");
            request2.ThinkTime = 1;
            request2.QueryStringParameters.Add("CategoryID", "14");
            request2.QueryStringParameters.Add("selection", "1");
            yield return request2;
            WebTestRequest request3 = new WebTestRequest("http://vsncts02/StoreCSVS/Static/ProductDetails1.htm");
            request3.ThinkTime = 6;
            request3.QueryStringParameters.Add("productID", "355");
            request3.PostRequest += new EventHandler<PostRequestEventArgs>(request3_PostRequest);
            yield return request3;
            WebTestRequest request4 = new WebTestRequest("http://vsncts02/StoreCSVS/Static/SearchResults1.htm");
            request4.ThinkTime = 4;
            yield return request4;
            WebTestRequest request5 = new WebTestRequest("http://vsncts02/StoreCSVS/Static/AddToCart3.htm");
            request5.QueryStringParameters.Add("ProductID", "355");
            yield return request5;
        }
        void request3_PostRequest(object sender, PostRequestEventArgs e)
        {
            // Wait for the configured number of web tests to complete this request
            // This call will wait for up to 2 minutes (120,000 ms) before timing out and throwing
            // an exception (the bool argument is whether or not to throw an exception when a timeout occurs)
            s_syncPoint.Wait(120000, true);
        }
    }
}

Page view tracker