Load testing in the cloud
Discover performance issues with your app before they impact your customers. Verify your app’s capacity for the next launch or marketing campaign using cloud-based load testing with Visual Studio Online.
How does cloud-based load testing make this easy?
- Quick: Get a performance lab in the cloud in almost no time with just one click. Scale up or down based on your needs.
- App performance insights: Get a full view into server and client diagnostics when you simulate real-life load patterns on your app.
- Free monthly usage: Get 20,000 user minutes free every month to load test your app.
If you already have existing load tests, or you know how to create them using Visual Studio, jump ahead to configure your project to run load tests in the cloud.
Download and configure a sample load test
Use Visual Studio 2013 Ultimate to create and run load tests in the cloud. If you don't have this or the latest Visual Studio 2013 update installed, download and install from here.
Download the sample load test project.
Extract the files and open the GettingStartedWithLoadTesting.sln.
Open the file SampleWebTest.webtest and replace the URL to use a web page for your app.
Save the solution.
Configure your project to run load tests in the cloud
If you're using the sample project, jump ahead to Connect to your Visual Studio Online account. If you have existing load tests that you created using Visual Studio 2010 or Visual Studio 2012, just open your existing project in Visual Studio 2013 first.
Edit the test settings file to configure your project to run your tests using Visual Studio Online. (Only load tests can be run in the cloud.)
You don't have to connect to Visual Studio Online with Visual Studio until you're ready to run the load test.
Connect to your Visual Studio Online account
Before you can run your load tests in the cloud, you must connect to your Visual Studio Online account. (If your team doesn't already have a Visual Studio Online account, create it first.)
Connect to your Visual Studio Online account from Visual Studio Team Explorer.
View the list of available team projects.
If you haven't connected to your Visual Studio Online account before, then you need to add it to your list.
Enter the name for your account: [youraccount].visualstudio.com.
If you're prompted to sign in to your Microsoft account, go ahead and do that.
Select the server from the list and then select your team project. Now you can connect.
Run and analyze your load test
From your solution in Solution Explorer, open the load test that you want to run.
Run the load test in the cloud using Visual Studio Online.
You can see when your test is in the queue for the service.
When your load test is ready to be run, the test status changes to: Acquiring resources. For a large test run this can take up to ten minutes while the service does the hard work of setting up machines and agents for you.
When the test is running, you can see the performance graph. Choose details to see any errors, warnings or other information related to your load test run.
You can also view the throughput graph.
After the load test is finished, you can download the report.
When it has downloaded, you can view the report.
The results for the completed test includes performance counter data, threshold violations, and error information.
Choose the detail view. By analyzing the step load pattern for users, you can identify the user count where your performance fails to meet your requirements.
Fix any performance issues in your application’s code and rerun the test. You can also refine your test to better simulate real-world loads by specifying various web performance test properties, load test scenario properties and run settings properties.
View your load test runs at any time
Open Load Test Manager to view past load test runs or currently running load tests. (You must have a web performance and load test project open to do this.)
Now you can view any load test runs started by any member of your team.
You can filter out the test runs that you see by state, date, or user who created the test run.
Get performance data from Application Insights
When you run load tests for your app using Visual Studio Online, you can also get performance data by using Application Insights. By correlating load and performance data together, you get more insight into any performance issues that you find. Then you can run a quick root cause analysis using Application Insights, too.
If you're using Application Insights with Visual Studio Online, continue with these steps. Otherwise, if you're new to Application Insights, or if you're using Application Insights with Azure, contact vsoloadtest@microsoft.com to get instructions.
Install Visual Studio 2013 Update 4, if you haven't already.
If you haven't set up monitoring for your app with Application Insights, do that first. You must use the same Microsoft account to monitor your app and run your load test with Visual Studio Online.
Open your load test in Visual Studio.
Add applications to get performance data from Application Insights.
Select the applications that you want to monitor and the counters that you want to view when your load test runs.
When your load test runs, you can view the performance data from Application Insights. There may be a delay of a few minutes before this data is displayed.
Application counters are correlated with the user load so that you can understand what issues may be causing any performance problems that you find.
Go to Application Insights if you need to do a more detailed analysis of any performance issue or to do a quick root cause analysis.
Q&A (Cloud-based load testing)
Q:How do I learn more about cloud-based load testing?
A:Watch this video, or check out the links on this blog here.
 
Q:Can I use cloud-based load testing to test any internal web site?
A:No. You cannot use a web site that is only available internally to your company. For example, an internal staging or preproduction deployment. However, you can open up the endpoint for your web site for limited IP addresses or users to work with cloud-based load testing. Or you can run a load test locally using Visual Studio.
Q:What are test agents and why do I need them?
A:Test agents simulate the load on your web site by creating virtual users. You might need to use more agents, depending on your load test and the number of virtual users that you require. Your agent count (total cores) can't exceed 100, and your ratio of virtual users to agents must be 25 or more.
For example, increase the number of agents that you're using, if you're getting status messages that an agent stopped working due to load, or if the downloaded report shows high CPU use for an agent.
If you run a test with 40 virtual users and 2 agents, the ratio is 20. This is too low, so increase the number of virtual users or reduce the number of agents. If you use the default value of 0, this ratio is not applied.
Q:Are there any limits on virtual users when running cloud-based load tests?
A:Yes. The limit is 20,000 virtual user minutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your Visual Studio Online account. The account will be charged for any virtual user minutes that you use above this limit each month.
Q:How can I run load tests locally and in the cloud from the same project?
A:You can have multiple test settings in a project. Add another test settings to the solution items folder. Now you can use one to run your tests locally and the other to run your load tests in the cloud. To switch between them, use the context menu to set the appropriate test settings to be active.
Q:Can load tests use other test types in their test mix besides web performance tests?
A:Yes, you can include unit tests and coded web tests. But coded UI tests are not supported with cloud-based load testing.
Q:Can I debug a load test while it's running in the cloud?
A:No. You need to run a load test locally to debug it.
Q:Can virtual users simulate pausing between test steps?
A:Yes, you can specify think times. Choose a scenario in your load test and edit the think time in the Properties view.
Q:When I download my load test report, where is it stored?
A:There is a local SQL express database where the downloaded reports are stored. You can change the default location if you want. Change the location for each user to the same database if you want to store all the reports together.
Q:Are there any load testing features that are not supported when you run a load test using cloud-based load testing?
A:These features are not currently supported:
- Goal-based load patterns
- Iteration-based runs
- Network mix property
- Warm up and cool down time
- Agent to Use in test settings - use the core count property instead
- SQL Trace properties in run settings
- IP switching
Q:Having problems running your load tests in the cloud?
A:Check out the latest troubleshooting advice.
Q:Where do I get more information about simulating real-world loads?
A:Find out more about how to specify web performance test properties, load test scenario properties and run settings properties.
Q&A (Get performance data from Application Insights)
Q:Can I view data from other application monitoring tools when I run load tests in the cloud?
A:No.
Q:Can I increase the frequency used to collect the counters?
A:No. This is currently a fixed frequency of 1 minute.
Q:I don't see any counters even after waiting a few minutes. What's wrong?
A:Go to Application Insights and check that you can view performance data for the application there. If there is data collected there, report your issue to vsoloadtest@microsoft.com. If there is no data, go here for troubleshooting help with Application Insights.
False