CLR Inside Out
Ensuring .NET Framework 2.0 Compatibility
If we learned only one thing about compatibility in the past few years, it is that compatibility is much more than avoiding breaking changes. On the Microsoft®.NET Framework and Visual Studio® teams, we do our part to ensure that the products we build are stable platforms that developers can truly rely on. At the same time, developers working on these platforms need to do their part to ensure that their applications can withstand just a little shaking. I'll take you on a behind-the-scenes tour of our compatibility efforts and discuss what we learned in the process.
What is Compatibility?
So what exactly is compatibility, and how many ways can an application be compatible? Well, first there's platform side-by-side (SxS) compatibility. This refers to the ability to have multiple versions of a platform installed and running at the same time, allowing each application to run on the version of the platform on which it was built. For example, on a machine running Windows Vista™ with versions 1.1 and 2.0 of the .NET Framework installed, apps built against the .NET Framework 2.0 will run against the 2.0 version, while any older apps will still run against the 1.1 version.
Platform-backwards compatibility is the ability for an application built on one version of the platform to run properly on later versions of that platform, while platform-forward compatibility means that an application built on one version of the platform can run properly on earlier versions of that platform.
With the .NET Framework 2.0 release, both side-by-side execution and backwards compatibility are supported. The 1.1 release supported forward compatibility as well, but too many features were added and improved in version 2.0 for it to be likely that you could develop an application built on the .NET Framework 2.0 that didn't use them. While it may not be a surprise that we no longer support forward compatibility, we did want to continue to support both side-by-side and backwards compatibility.
Side-by-Side Execution and Backwards Compatibility
On the .NET Framework and Visual Studio teams, the most important of our compatibility goals mandated that installing the .NET Framework should not break existing applications, regardless of the fact that most applications would be relying on side-by-side execution in this scenario and simply run against the version they were built on. Next in order of importance was that we wanted to ensure that applications would run properly if only the latest runtime were installed. There was a bit more leeway here because we expected side-by-side to be the typical configuration, but we also knew that plenty of version 1.1-based applications would be running on version 2.0.
Finally, we wanted applications built on Visual Studio .NET 2002 or Visual Studio .NET 2003 to be easily upgraded to use Visual Studio 2005. We decided that a developer should be able to convert a large, multiproject solution in a couple of hours.
Of course, no matter how careful we were going to be, there would always be some applications that would break when run on a later version of the Framework. When trying to determine if the value of a breaking change outweighs the impact of the break, we would look at these areas of impact, in order: security, standards compliance, reliability or determinism, and correctness.
At the same time we looked at the following criteria for determining the impact of the break: Does it break side-by-side execution for some applications? Will many applications break in a scenarios using only version 2.0? Will it be easy to determine the problem and fix the application?
As you might expect, sometimes applications break for reasons beyond our control. For example, the change that causes the most applications to break with each release is the increment in the version number. There are also many applications that will fail because they have race conditions that will be exposed once the performance characteristics of the runtime have changed. Still other applications will fail because they took dependencies on the runtime that are far beyond the scope of our API's public contracts (for example, using private reflection to read and write private members of Framework classes). These are the types of application breaks that make our guidance on writing compatible programs so vital for application developers and where it's their turn to do their part to ensure continued compatibility.
For the releases of the .NET Framework 2.0 and Visual Studio 2005 we decided that we needed to be 100 percent compatible with the previous releases. We assembled a set of applications of various types, sizes, and complexity to test against the latest builds. On day one, build one, our pass percentage was very high but as check-ins were made the number started to drop.
Initially, most of the failing applications were blocked by simple bugs—the type that tend to creep in at the beginning of a major development cycle. As these bugs were fixed and others opened, our passing percentages began to trend upwards. We soon reached some of the first intentional breaking changes of the product and it became clear that our goal of 100 percent compatibility was in direct conflict with some of our plans for new features. Thus the Developer Division Compatibility Council (DDCC) was formed.
DDCC was a central group of three senior members of teams from across the division whose job was to make the trade-offs between compatibility and other quality metrics. We retained our target compatibility goal of 100 percent but added a * next to the number with the footnote: "except for DDCC-approved breaking changes." After this, when teams wanted to make a breaking change or when one was discovered, the team would go to DDCC for a decision. The pass percentages for our applications continued to rise and were very high by the time we shipped Beta 1.
Soon after we shipped Beta 1, we received MSDN® Feedback reports indicating that the high passing percentages we saw in our test lab were not realized in the real world. So we released an instrumented build of Beta 1 in the hope that developers would use it and tell us which of the .NET Framework APIs they were depending on. This would help us determine the possible impact of our breaking changes. We did everything we could to publicize this release. Still, in the end we received approximately five uploads of code coverage information. It was clear we needed a new way to reach customers.
As a result, we held the first of many Compatibility Developer Labs. Here we discovered that when configured to run on the .NET Framework 2.0, 100 percent of the version 1.0 and version 1.1 applications tested failed. Except for two Microsoft Office Add-ins, every single application passed in the default side-by-side configuration (with both the .NET Framework versions 1.1 and 2.0 installed).
With the realization that our previous success metrics were insufficient, we took a step back and regrouped. We retained our goal of 100 percent passing except for DDCC-approved breaking changes. In addition, we added absolute metrics detailing what percentage of various types of apps must pass in each of the compatibility scenarios, regardless of reason for failure.
Next we started an extensive search for external applications from many different channels. In a couple of months we expanded our internal test suite from fewer than 50 applications to more than 250 that included everything from Microsoft server applications, to small Web downloads, to enterprise-level inventory and personnel software, and even two robotics applications. With this expanded suite we were able to validate our compatibility against a wide range of applications and could finally identify the issues that were impacting multiple applications.
In addition to expanding our in-house suite of applications, we expanded our first Compatibility Developer Lab and started bringing more developers on site so that we could assist them in testing their applications on the latest builds. We were able to test apps we would never be able to bring in-house and get results for scenarios that we usually can't test.But some apps couldn't be moved from one environment to another and some customers couldn't make it to Redmond. So we went to them.
In the end, these efforts had a profound impact. By expanding our internal coverage, bringing customers to us, and visiting them directly, we were able to find classes of compatibility issues that we had simply never encountered before. The result was that the vast majority of applications will not be impacted by the installation of the .NET Framework 2.0, most applications will work just fine on machines that only have the latest version, and, with the late-breaking changes to ASP.NET project migration due to feedback from our labs, most developers will have no problems migrating large, multiproject solutions, from Visual Studio .NET 2003 to Visual Studio 2005, in the course of an afternoon.
For a look at two actual breaking changes and how we decided to handle them, see the case studies sidebars (Case 1: Visual Studio Tools for Office 2003 and Case 2: Enterprise Library). They were selected because we expect that they are really the only two compatibility issues that are going to impact more than a few isolated apps. They both also have interesting stories behind them and will impact developers and their applications in different ways.
Send your questions and comments to email@example.com.
is a Program Manager on the CLR team at Microsoft and, among other things, is responsible for application compatibility. You can contact him at firstname.lastname@example.org