Performance Testing Guidance for Web Applications

J.D. Meier, Carlos Farre, Prashant Bansode, and Scott Barber
Microsoft Corporation

September 2007

It’s hard to imagine anything than is considered a more arcane art than performance tuning – unless perhaps it is performance testing. 

If you were to go door to door between groups just within Microsoft you would find many different approaches with various different degrees of quality or success.  Pretty much everyone will vow that their approach is certainly the one that is best for them – except maybe an honest few, who might say something more modest.  Some have good reason to be confident because they really have studied the space very well.  In my own experience at least, the situation is not altogether different outside of Microsoft than it is inside where I do my work.  It’s a mixed bag, on a good day.

If I had to describe the most common problem I see in this space with one word it would imbalance.  There are many aspects to testing and teams tend to unduly focus on one or another and then sometimes get blindsided by the ones they missed.  Perhaps they’re only thinking about throughput – what about consumption?   Perhaps only latency – what about smooth delivery?  Perhaps only cost -- what about scalability? 

You get great performance by balancing the key factors, considering them in your designs and then tracking them carefully.  So perhaps the greatest service that a book like Performance Testing Guidance for Web Applications can provide to you is a broader understanding of what all the factors might be so that you have an excellent menu of considerations to choose from in your testing plan.  Luckily, that is just what you’re going to get.

The Guidance that follows provides a great survey of the most important considerations:  From how to understand and quantify your desired end user experience, how to choose key resources for study, to advice on summarizing results in a statistically meaningful way, and how to fit these practices into different software lifecycles.  And even though the focus is squarely on web applications, the teachings are actually much more general and can easily be applied for many different kinds of applications.

Great engineering comes from creating predictable results at predictable costs. In fact, I like to say that if you’re not measuring you’re not engineering.  This volume will provide you with the performance testing fundamentals to give you the ongoing metrics you need to do great engineering.

Rico Mariani
Chief Architect of Visual Studio
Microsoft Corporation
July, 2007

Rico Mariani began his career at Microsoft in 1988, working on language products beginning with Microsoft® C version 6.0, and contributed there until the release of the Microsoft Visual C++® version 5.0 development system. In 1995, Rico became development manager for what was to become the "Sidewalk" project, which started his 7 years of platform work on various MSN technologies. In the summer of 2002, Rico returned to the Developer Division to as a Performance Architect on the CLR team.  His performance work led to his most recent assignment as Chief Architect of Visual Studio. Rico's interests include compilers and language theory, databases, 3-D art, and good fiction.