Share the load
Report Visual Studio Team System Load Test Results Via A Configurable Web Site
Wen Ding
This article discusses:
- Using the Visual Studio load test tool
- Load test reporting Web site
- Architecture of the test reporting site
- Using and extending the test reporting site
|
This article uses the following technologies:
Visual Studio 2005, Team System, SQL Server 2005
|
Code download available at:
LoadReporting2006_06.exe
(332 KB)
Browse the Code Online
Contents
V
isual Studio 2005 Team System includes a newly designed load test tool. Formerly code-named "Ocracoke," this new tool provides many exciting features you can use for performance and stress testing your Web sites, Web services, and other server components. And, combined with its handy reporting capabilities, the load test tool provides some powerful options for sharing and managing test results.
The load test tool executes performance test runs and can be configured to log data into a central SQL database. After each test run, you can view collected test data, such as performance counter values, test error messages, and the test execution load, and then analyze the test data from within the Visual Studio® 2005 UI.
I find it particularly helpful to create performance test reports in a summary view, showing key performance results like latency numbers, test load, server resource utilization, and any errors encountered during execution. These performance reports can be viewed with a Web browser from other machines that might not have Visual Studio installed. Because the load test tool can store test data in a SQL Server™ database, users can connect to the database directly and dig into the test data repository to create a variety of customized performance test reports. In this article, I'll introduce one approach to creating a customized performance reporting Web site from test data generated by the load test tool.
The reporting site I discuss in this article (shown in Figure 1) provides two main features. The first is an ability to show reports of performance test runs that can be viewed from any Web browser. The reports can also be exported to Microsoft Excel® or Office Web Components-based HTML documents for e-mail delivery. For each test run, the site displays several useful reports including latency numbers for both test cases and transactions; summary and detailed views of performance counter logs that show the amount of server resources consumed (such as CPU and memory usage); test load information including the number of requests per second; and error messages.
The second feature provides very handy management functions. The number of performance test runs grows over time, especially when your team has been performance testing multiple projects. This could result in hundreds of different performance test runs being stored in the database, making it difficult to search for information or clean up any obsolete test runs within Visual Studio. This sample Web site lets you easily filter test runs by, say, scenario name or date range, and then remove any unneeded test run data from the database. Other useful management functions include editing project and scenario names and adding analysis comments.
Using the Load Test Tool
When you develop a high-traffic Web site, a Web service, or some other server components, it's important to know how much load your site or service can handle. A performance test can help determine whether the product meets your performance goals. You can use a load test tool like the one in Visual Studio 2005 Team System to simulate multiple users simultaneously sending requests to the service. Although I won't go through all the exciting new features of that tool, I will describe some basic features that will help you understand how the test reporting site works. You can find more information on working with the Visual Studio 2005 load test tool at
msdn2.microsoft.com/ms182561.aspx.
The load test tool is fully integrated into Visual Studio 2005 Team System, so you do not need to search for and launch it separately. The Load Test Wizard lets you easily create a load test project. To start a performance test run you create unit test cases covering your desired performance test scenarios, and then create a load test project. At this point you can choose any unit test cases that you want to run performance tests against, and add them to the load test project. The Load Test Wizard guides you through all the necessary steps including creating multiple performance test scenarios with different run settings. After these tasks, you can run a scenario for performance testing.
There are two test case types that can be added to a load test project: Web page tests and unit tests, including any functional code or Web service calls. To create a Web page load test, you use the Web Page Recorder to capture the navigation and user input performed in the Web browser. The recorded Web requests, post data, query strings, and cookies can then be replicated by the load test tool. The Load Test Wizard will guide you through all the replay settings: how many simultaneous users, the mix of browser types, the run duration, thinking time between Web requests, the mix of different network bandwidths, what types of performance counters to collect, and so on.
For load tests on unit test cases, such as Web service calls or calls to server components, you can create a load test project and add unit cases to it. The same Load Test Wizard guides you through settings that are similar to those mentioned previously. You can choose different stress patterns, including consistent load, step-up load, and so on. Load tests can be done on a single test client machine, or initiated from a set of remote machines (called a "rig"). A controller machine is used to control the test run on the rig, collect all the performance test data from the remote clients, and place all the data into the results database.
The load test tool is able to collect various types of test data from remote servers and log this data into the database. For performance test reporting, it is important to know what kinds of test data are collected after the load test run. There are four types of information you will typically put in the reports:
- Latency for each type of request
- Throughput information, or how much load the server received (for example, requests per second)
- Server-side resource utilization (such as CPU usage, memory consumption, and so on)
- Test run configurations (such as test duration, number of simultaneous users, and so on)
There are other useful types of information that can be logged in the results database. These include any error messages encountered and SQL Server trace information that can help you find which query or stored procedure is the slowest during the test run.
The Load Test Reporting Site
My load test reporting Web site is designed to provide a number of useful capabilities that can help you run tests and create reports. First of all, it has a hierarchical structure that makes it easy to navigate between test runs. The site organizes test runs by date range—an approach that makes it easier to find and manage tests. It allows for easy modification of important information like project and scenario name, test description, and analysis comments. And my reporting Web site offers a variety of test reports, including a summary report, a detailed performance counter report, and a Web page performance test report. (Note that my sample site uses Scott Mitchell's RoundedCorners Web control, which you can read more about at
aspnet.4guysfromrolla.com/articles/072804-1.aspx.)
On the left side of the home page in Figure 1, a tree view navigation control lists all the load-tested projects in the database. When you click on a specific project name in the tree view, all the scenarios and test runs for this project are displayed in the right panel. The relationship between a project name (the LoadTestName field in the database tables), scenario names, and RunIds is like a branching tree-structure. One project can contain multiple scenarios. And for one test scenario, multiple test runs can be executed. Under the tree view control, there is a dropdown list control that is used to filter the number of test runs displayed. To limit the number of test runs displayed on the page, you can specify dates to display only test runs that happened on those days.
.gif)
Figure 1 Load Test Reporting Site
On the right side of the home page, the Results column provides three links for each run: Summary (which displays a test summary report, as shown in Figure 2), LoadChart (which displays a user load chart), and PageResults (which shows results for a Web request test). The Edit links allow you to modify the project name, scenario name, test run description, and analysis comments for a specific run, and the Delete links let you remove all the logged test data for a specific run from the database.
.gif)
Figure 2 Test Summary Report
Report Site Architecture
The design architecture used to build the report Web site is shown in Figure 3. The Web pages are built with ASP.NET Web controls that are bound to data from the results database. For all test reports, SQL Server Report Definition Language (RDL) is used to pull data from the database and create report pages. Several new stored procedures and views are created in the database and used by these Web controls. My site takes advantage of many features that SQL Server Reporting Services provides, such as exporting reports to other formats.
.gif)
Figure 3 Report Site Architecture
Even with this reporting site, you still need Visual Studio 2005 Team System to run load tests and configure the test run to log raw data into a centralized SQL database. However, by using this site architecture, you can easily customize the design of your own reporting pages by either modifying the sample report designs or creating your own reports.
The sample solution includes an ASP.NET site- setup project, WebSetup_PerfReport, from which you can install the Web site on any host server with IIS and ASP.NET 2.0. SQL Server RDL files are provided to build the test reports. But some setup steps are necessary before you can compile the sample projects and install the reporting Web site. (See the code download for details.)
After you have set up the reporting Web site to pull test result data from the SQL database and create test reports, you can test your reporting site and start writing Web tests or unit test cases and add them to a load test project. After running the load test project, check whether the test run shows up in the reporting site. As mentioned earlier, you can view the test results right after the test run, using Visual Studio 2005. The advantage of the reporting site is that you can get a summary report and detailed performance test reports simply by using a Web browser on any machine.
To write your own test report by accessing test data directly from the load test database, you need to understand the basic table structure, stored procedures, and views. Because the load test tool internally logs data into the tables, changing table design or structure could potentially break the test tool. Therefore, you need to adapt your approach to table structures required by Visual Studio. For example, in the sample reporting site, test runs for all test projects are organized in a tree view with a hierarchical structure like project name/scenario name/run ID. But the LoadTest database only contains the fields ScenarioName and RunId. Fortunately you can use the LoadTestName field to serve as the project name for reporting needs.
An example of using a customized stored procedure is shown on the right-hand side of the main page, shown in
Figure 1, where the ASP.NET DataGrid control binds to test data returned from the following stored procedure:
CREATE PROCEDURE Prc_SelectLoadTestSummary
@LoadTestName nvarchar(255)
AS
SELECT ScenarioName, LoadTestRunId, Description, StartTime,
EndTime, RunDuration, Outcome
FROM dbo.LoadTestSummary
WHERE LoadTestName = @LoadTestName
Load Test Run Management
Typically, you will conduct a number of performance test runs including several performance test scenarios for each project. This could result in a large number of test runs logged into the results database. Some of the results may not be useful, and you may want to remove them from the database. For more relevant test runs, you probably want to add analysis comments to summarize your observations and present them in test reports.
A nice feature of the new load test tool is that, for each performance test run, you can put your own analysis comment in the comment field of the table LoadTestRun. The comment is stored in the database after each test run, where you can analyze data and put summarized findings about the performance test run into it.
The page shown in Figure 4 was designed to allow users to modify test information through the reporting site. This page can be launched by clicking the Edit link on the main page. You can change the project name by entering a new one or choosing any existing project name from the dropdown control. You can also edit the test scenario name, test run description, and analysis comments in the same page. This function is especially useful when a certain test run is identified as the final test run and you want to share the results with your team. Once you click the Update button, the database will be updated with the new information.
.gif)
Figure 4 Updating Test Run Information
If you find some test runs are not worth keeping in the database, there are several ways to remove them. One way is through the reporting site, where you can simply click the Delete link on the main page. The link internally calls a stored procedure Prc_DeleteLoadTestRun by passing a test RunId parameter corresponding to the clicked DataGrid row.
You need to pay attention to the delete action from the reporting site because any user accessing the page could accidentally remove test runs by clicking Delete. There are several things you can do to prevent this, though. For example, you can add authentication logic to restrict access to critical pages that offer the delete or edit actions.
You can remove multiple test runs at the same time by using your own stored procedure. One example of such a stored procedure is to remove test data for all test run IDs in a range (Prc_DeleteLoadTestRunFromIds):
CREATE PROCEDURE Prc_DeleteLoadTestRunFromIds
@LoadTestRunStartId int,
@LoadTestRunEndId int
AS
BEGIN
DECLARE @LoadTestRunId int
SET @LoadTestRunId = @LoadTestRunStartId
WHILE @LoadTestRunId <= @LoadTestRunEndId
BEGIN
EXEC Prc_DeleteLoadTestRun @LoadTestRunId
SET @LoadTestRunId = @LoadTestRunId + 1
END
END
GO
For example, to remove test results for ID 12 through 20, you'd simply run the following:
EXEC Prc_DeleteLoadTestRunFromIds 12, 20
SQL Server Reporting Pages
From the main test report page you can open three reporting pages for a test run by clicking Summary, LoadChart, or PageResults. The code behind the DataGrid control redirects the request to the corresponding reporting service page as shown in Figure 5. The three reports are designed with LoadTestSummary.rdl, PagePerformanceOverTime.rdl, and RequestPerformanceDetails.rdl in the sample project.
Figure 5 Redirect to SQL Reporting Pages
private void DataGrid1_Item(object source,
System.Web.UI.WebControls.DataGridCommandEventArgs e)
{
try
{
string runID = e.Item.Cells[1].Text;
switch(e.CommandName)
{
case "Results" :
Response.Redirect(
"http://MySqlServer/ReportServer?" +
"%2fOcracokeReport%2fLoadTestSummary&" +
"TestRunId=" + runID, false);
break;
case "LoadChart" :
Response.Redirect(
"http://MySqlServer/ReportServer?" +
"%2fOcracokeReport%2fPagePerformanceOverTime" +
"&TestRunId=" + runID, false);
break;
case "WebRequests" :
Response.Redirect(
"http://MySqlServer//ReportServer?%2fOcracokeReport" +
"%2fRequestPerformanceDetails&TestRunId=" +
runID, false);
break;
case "Edit" :
Session["RunId"] = runID;
Response.Redirect("home.aspx", false);
break;
}
}
catch (Exception ex) { Debug.WriteLine(ex.ToString()); }
}
A key area of performance test reporting is system resource utilization on each test server. In the report summary page of Figure 2, you'll see there is a section called System Resource Usage. In this section, several important performance counter categories are reported. If you are interested in adding more pre-selected performance counter instances or categories in the summary report, you can modify the SQL query in the data source for report LoadTestSummary.rdl as shown in Figure 6.
Figure 6 Performance Counter Query
SELECT *
FROM LoadTestComputedCounterSummary
WHERE loadTestRunId = @TestRunId AND
((CategoryName = ‘Processor’ AND
CounterName = ‘% Processor Time’ AND
InstanceName = ‘_Total’) OR
(CategoryName = ‘Memory’ AND
CounterName = ‘Available MBytes’) OR
(CategoryName = ‘System’ AND
CounterName = ‘Context Switches/sec’) OR
(CategoryName = ‘.NET CLR Exceptions’ AND
CounterName like ‘%Excep%’) OR
(CategoryName = ‘.NET CLR Memory’ AND
CounterName = ‘% Time in GC’) OR
(CounterName = ‘Requests/Sec’ AND
CategoryName Like ‘%ASP%’) OR
(CounterName Like ‘%Queue%’ AND
CategoryName Like ‘%ASP%’) OR
(CategoryName = ‘Web Service’ AND
CounterName = ‘Current Connections’) OR
(CategoryName Like ‘%Network Interface%’ AND
CounterName = ‘Bytes Total/sec’))
ORDER BY MachineName
In the System Resource Usage area, you can also click on each performance counter category name and jump to a more detailed report for the performance counter values in that category on all the test servers. This function lets you dig into more performance counter instances inside that category by showing the values for all the servers involved. For example, by clicking the System link, you get a detailed report page for system resources. The report page shows all the performance counter instances under the System category for all the involved test servers. This is especially useful when you want to analyze more detailed performance counter data for a specific performance counter category. This report is in SystemResourceUsageDetail.rdl.
If you have WebRequest in your load test project for Web page performance testing, you can view the Web page result report by clicking the PageResults link. This brings up a report page that lists all the test information about a certain Web request. You get multiple rows if you are performance testing multiple Web pages. The RDL file for this Web request report is RequestPerformanceDetails.rdl.
After you run a load test, the response time and latency numbers for test cases and transactions are shown in average values. The data in the Test Case column represents the elapsed time for any function in your code marked with the [TestMethod] attribute, as shown here:
[TestMethod]
public void SubmitEventWithEmail() {
SubmitEvent(false);
}
[TestMethod]
public void SubmitEventWithPuid() {
SubmitEvent(true);
}
Because there are two functions marked with the [TestMethod] attribute in this code, the latency data or response time for both of these functions is reported in the Test Case column. The response time represents the overall latency for the entire function.
It is important to measure latency for any sub-functional calls inside a large function. This can be implemented in your code by using BeginTimer and EndTimer functions to measure time being spent on any given functions. An example of this is shown in Figure 7. In this code, latency data will be collected for any functions surrounded by BeginTimer and EndTimer. The latency number is reported in the Transaction section of the page results.
Figure 7 Measuring Transaction Latency
private void RunSinglePath()
{
m_testContext.BeginTimer("GetChannels");
string ch = _ch.GetChannels("/", _local, -1);
m_testContext.EndTimer("GetChannels");
m_testContext.BeginTimer("Add All - " +
Items.Length.ToString() + " items");
AddAll(false, false);
m_testContext.EndTimer("Add All - " +
Items.Length.ToString() + " items");
}
private TestContext m_testContext;
public TestContext TestContext
{
get { return m_testContext; }
set { m_testContext = value; }
}
Going Further
Once you are able to build your performance test reporting site, there are many things you can do to improve it. One report you can add is a SQL Trace report. If you enable SQL Trace Tracking in your load test project, you can get execution time information for SQL query commands or stored procedures in the LoadTest database. Based on SQL Trace data from a sample test run, you can create a report to display stored procedures ordered by latency, allowing you to point out the slowest stored procedure in the test run.
A side-by-side comparison report for two performance runs is also very beneficial, as it lets you see if a new version of a build has better performance than older versions. On a single Web page, you can display the performance differences (for, say, latency or CPU usage) between two versions. This way, you can instantly compare the performance of the two runs and find out the areas that require further investigation.
The reporting site I've discussed in this article displays average values for response time and performance counter. If you prefer performance test data represented in percentiles (rather than averages), refer to
www.seanadams.com/95 for more information. Percentile data does have the benefit of providing statistical information.
With the default configuration of the load test tool, the percentile data is not collected. In order to enable the percentile calculation, you must change the property Timing Details Storage of Run Settings in the load test project. For more information about this setting and other useful techniques, visit Bill Barnett's
blog. The percentile data is logged into a table called LoadTestTransactionSummaryData. You can modify the SQL reporting design schema to include percentile information on report pages. Note that enabling this will add extra data to the database and more disk space will be consumed.
Finally, unlike with functional test cases, identifying a performance test run as pass or fail doesn't always make sense. Sometimes you just want to collect benchmark performance numbers. In other cases, however, you probably have performance requirements in terms of acceptable and unacceptable latency or CPU usage. If the test run performance falls outside of the acceptable range, the test is essentially a fail. To identify whether a performance test run meets performance requirements, you can modify the current reporting design schema and Web controls to add this kind of result information to a specific test run.
Conclusion
Visual Studio 2005 Team System has integrated a handy new load test tool that allows you to run customizable tests and collect critical data. But other capabilities, such as the test reporting Web site feature, let you dig deeper and get more out of the built-in load testing. With a firm grasp of the test reporting Web site functionality, you can more easily share test results, analyze data, and manage stored test results.
Wen Ding works for Microsoft where he conducts software performance testing and optimization for many projects used on microsoft.com sites. Wen can be reached at
dingwen@microsoft.com.