Microsoft IT prepares LOB apps for Windows

Technical Case Study

May 2015

Microsoft IT takes an agile approach to line-of-business (LOB) application compatibility testing. Their approach enables them to significantly reduce time and labor requirements to test their application portfolio by reducing the number of applications tested. This, in turn, facilitates upgrades to new technologies such as the latest version of Windows or Microsoft Office.

Situation

Application compatibility is a concern anytime an organization deploys new software into an existing computer environment. Whether the organization is deploying a new operating system, installing a new web browser, or updating their productivity suite, it must ensure that the applications that users need to perform their daily tasks continue to function properly. Microsoft IT shares this challenge by retaining responsibility for maintaining and deploying more than 2,100 internal line-of-business applications in an environment of mixed and changing operating systems, web browsers, productivity applications, and development tools.

Microsoft IT defines an LOB application as an application used to perform a key business function. Within the Microsoft IT LOB portfolio, 93 percent are web-based applications.

The primary reason Microsoft IT tests LOB applications is to ensure that the internal LOB applications are compatible with new operating system (OS) platform versions (Windows and associated versions of Internet Explorer) and the Office productivity suite before deployment to the 1.2 million devices that connect to the Microsoft network worldwide. The application compatibility testing process is designed to reveal issues with the key scenarios for which the LOB application was created, such as payroll and expense reporting, procurement, hiring and resource management, or sales commission. It also targets scenarios where an application has only a small integration aspect, as in the case with an Office release.

Because of the broad variety of LOB applications that internal employees use, Microsoft IT doesn’t have the time or people power to test all LOB applications during short product release cycles. To overcome the constraints required to test their application portfolio, Microsoft IT needed to take an agile approach and to make smart business decisions around application compatibility testing based on acceptable levels of risk to business continuity.

Solution

Microsoft IT developed a sample-based approach to testing their LOB application portfolio. Business Process Units (BPUs)—such as Finance, HR, and Sales—identify their top business-critical applications, and Microsoft IT focuses on testing those applications. By conducting a successful test pass on those key applications, Microsoft IT gains confidence that the other applications with similar design, shared code dependencies, and underlying technology will also pass, if tested. Since these sample applications are typically the most complex, they serve as adequate proxies for the remainder of the LOB portfolio. Microsoft IT is able to test approximately 15 percent of the total applications in their LOB portfolio and to gain a reasonable assurance of compatibility for all.

Planning

Understanding the applications present in an environment and identifying the subsets that are business-critical are key first steps in testing for application compatibility. This information enables you to build a test plan and test matrix of the type and level of testing needed for each key application.

Application portfolio management

Microsoft IT maintains a database of all applications deployed in the organization for general use. They gain insight into those applications using an internal application portfolio management (APM) tool to track a wealth of historical application data. This enables Microsoft IT to identify which applications are most likely to provide key indicators to signal likely issues with other, similar applications.

 

Anytime a new application is created for internal use, it must be entered into the portfolio management tool. Microsoft IT further requires that, at a minimum, the person responsible for the application enter the basic identifying information for the application, such as the name and contact information for the person(s) responsible for the application’s compatibility impact. Typically, that contact person will be the person who is dedicated to testing the LOB applications for that BPU.

For each LOB application, Microsoft IT tracks:

·     Application name. The name by which the application is known in the network environment. The application name is useful when Microsoft IT contacts the product group to describe an issue.

·     Estimated number of users. This item is an estimate on the part of application owners. The main importance of this information is to help assess the impact on the company if the application fails. This is also an important criterion for selecting applications to be part of the key test group.

·     Whether the application is a non-Microsoft application. Because Microsoft IT has little control over the development of non-Microsoft applications and because Microsoft customers likely use similar third-party applications, these applications receive a high priority in testing.

·     Compatibility trending data. Microsoft IT uses the test results to analyze compatibility trends over time and to generate reports. Tracking the results over time enables more accurate predictions of compatibility issues and more focused testing. Applications that do not present any failing marks during testing may be bypassed in future testing.

·     Contact information. This is a critical data item for testing program success. Having accurate contact information enables Microsoft IT to pass along vital information efficiently. Without this information, Microsoft IT would lose valuable time locating the responsible party.

 

Tracking this information reduces the time and effort required in subsequent testing and helps identify which applications are at risk for problems.

The scope of information can include application information such as: cost to the company per day if an application is unavailable, which user group/segment is most affected by an application outage, any special hardware requirements for an applications, whether the application runs on a virtual machine (VM), and so on.

Regular audits of the LOB portfolio are performed to determine changes in criticality, application outage impacts, and technology dependencies. Likewise, Microsoft IT evaluates their portfolio to determine:

·     Applications that can be retired as they are no longer used or are duplicates of other applications.

·     If a newer version of an application makes more sense. For third-party apps, identify if the vendor has a newer version available. If developed in-house, evaluate if the time and budget required to constantly patch the application makes sense or if the application should be rebuilt to modern standards to eliminate future compatibility issues.

 

The better you understand your portfolio, the more information you can track in your APM. And the more accurate that information is, the more successfully you can target applications for testing.

Application targeting

Microsoft IT maintains lists of applications that the BPUs have designated as primary and secondary.

·     Primary. Critical applications required for business continuance, such as payroll and expense reporting, procurement, hiring and resource management, sales commission, and so on. Testing of primary applications ** is required during all test cycles.

·     Secondary. Less-critical key applications that the business depends on and that are identified through voluntary participation from internal groups. Testing of secondary applications is voluntary and may occur at any point during the test cycle.

 

Microsoft IT maintains a list of approximately 300 applications for the key priority test group—107 are primary for Windows, 108 for Internet Explorer, and 93 for Office. These applications remain crucial to business continuance. Microsoft IT cross-references the applications by technology dependency within this group. They ask individual business groups within the company to volunteer information about the importance of the LOB applications that they use. The information enables Microsoft IT to make better decisions on the relative priority of the applications. Microsoft IT also uses historical data from past testing cycles to identify applications that had issues in the past. For example, OEM Licensing Wizard and internal tools such as Performance@Microsoft (our internal review tool), MS Market, and MS Expense that were dependent on specific product features, such as Office file formats, were impacted with every new release of Internet Explorer.

Applications that consistently break with each new upgrade are included in the key test group. These applications are further evaluated to understand why they break and what updates are required to make them more resilient. Tracking this information helps Microsoft IT focus their testing on a much more manageable subset of the application portfolio.

Testing

The starting point for any accurate assessment of application compatibility is an understanding of the testing matrix, it defines the scope of which applications must be tested. For example:

·     Prior to releasing a new version of Office, determine what versions of Windows and Internet Explorer require compatibility validation.

·     Before releasing a new version of the OS, determine which types of device—such as desktops, laptops, tablets, and smartphones—are used to access the key LOB applications.

 

Microsoft IT maintains a fairly tight n (current version), n+1 (new version), and n-1 (previous version) support environment. For example, at the time this paper was written, n is currently Windows 8.1, n+1 is Windows 10, and n-1 is Windows 7 Service Pack 1 (SP1). Maintaining a tight support environment reduces the size of the testing matrix for internal, pre-release software (dogfood) testing. Internal application owners, especially those with applications hosted in the extranet, still need to know all the versions of Windows, Internet Explorer, and Office that their external users are likely to have.

Environment

To minimize cost and effort associated with the testing process, Microsoft IT leverages desktop virtualization and virtualization management features available in Windows Server 2012 R2 and Microsoft System Center Virtual Machine Manager 2012 R2. Using dynamically provisioned virtual machines (VMs), Microsoft IT tested LOB client applications on pre-configured virtual desktops. This provided a higher level of control and efficiency by providing identical configurations for all testers, eliminating the need and time required to build out physical machines. Issues found are attributable only to the new software, as all of the other software components installed were known to be compatible from previous testing programs.

Microsoft IT uses an open, self-service testing model where testers can test whenever they have time instead of within a pre-defined testing window. This is especially important for BPU participation in dogfood testing as it is not part of their daily work. No BPU has a dedicated dogfood test team, rather they fit dogfood testing in where possible.

In the past, testers were required to submit VM requests via email and then wait for notification when their pre-configured VMs were available, delaying the testing process. With the open testing model, based on System Center Virtual Machine Manager technology, testers can provision one or more VMs using templates through a self-service portal. Testers can interact with the VM as soon as the creation process completes. This change vastly reduced lost testing time, eliminating the need for testers in remote locations to wait for their requests to be completed by Puget Sound-based support personnel.

Starting with the Windows 8 and Internet Explorer 10 deployment, testing moved from a team-owned, dedicated application compatibility lab to a centralized service provided by Microsoft Agile Labs. Agile Labs is a shared cloud and data center management platform that enables internal Microsoft customers to efficiently manage their environments, server infrastructure, and client devices. By leveraging Agile Labs, Microsoft IT was able to significantly increase testing capacity, from 200 VMs in their own test lab to more than 1,000 VMs using the Agile Labs. This increase in capacity enabled Microsoft IT to accommodate more testers from across Microsoft and provided the ability to supply client configurations for multiple testing programs simultaneously.

On average, Microsoft IT requires about 200 VMs per test cycle and can conduct all the LOB testing required to support the product groups with less than 500 VMs. Having the additional capacity available through Agile Labs provides Microsoft IT with the ability to scale up their service offering.

Server component	Description
Processor	Two quad-core processors (2.66 GHz or higher)
Random access memory (RAM)	48 gigabytes (GB); supports 20 VMs concurrently with 2 GB of RAM dedicated to each VM and 8 GB reserved for the OS
System drives	148 GB; two 10,000-RPM serial attached SCSI (SAS) drives in redundant array of independent disks (RAID), one for the OS
Data drives	1.2 terabytes (TB); six 300 GB 10,000-RPM SAS drives in RAID 5 for VMs where each VM has a 50 GB dynamic virtual hard drive limit
Network adapter	On-board adapter is sufficient (two total network connections; one for VM traffic and one for transferring files to server)
Power supply	Redundant power supplies
Operating system	Windows Server 2012 R2
Virtualization software	Microsoft System Center Virtual Machine Manager 2012 R2 (installed on SCVMM host server)

Table 1. Typical server configuration for hosting the VMs used during Windows 8.1 and Windows 10 LOB testing.

Each server with the specifications shown in Table 1 can support 20 VMs concurrently, with one VM per user. If multiple users share a VM, performance suffers.

Teams

Two teams perform testing, using the same scripts and processes for application compatibility testing on the main Microsoft platforms of Windows, Internet Explorer, and Office.

·     A dedicated test team (referred to as the embedded testing team) has testers dedicated to a specific BPU, testing only their applications.

·     A BPU testing team that leverages testers in various BPUs of Microsoft IT to test LOB applications.

 

Having the agile test environment and the embedded testing team, Microsoft IT can change testing at a moment's notice as only the VM configuration requires changing to repurpose a scheduled test pass. This makes it much easier for BPUs to participate without significantly impacting their LOB application feature development schedules.

 

Process

The testing schedule aligns with the deployment goals of Microsoft IT, including:

·     V-next product milestone deployments:

o  Developer Preview (alpha): deployment to a small number of internal users

o  Consumer Preview (beta): deployment to a medium number of internal users

o  Release Preview: deployment to a medium number of internal users

o  Release to Manufacturing (RTM): full-scale deployment of internal release of the finished product

o  General Availability (GA): finished product released to the public.

·     Product updates

 

Microsoft IT enables the internal (dogfood) use of our new products by conducting LOB application compatibility testing prior to the deployment of the new release to internal users. With any deployment, even if only to 100 or 500 people, some amount of application compatibility testing occurs.

·     For a minor release, typically a service pack release, the embedded testing team performs a quick, one week test pass to verify there isn't anything substantially wrong happening.

·     For a major release or major deployment, a full Microsoft IT test pass is required. All BPUs and application teams are involved, at least for testing the critical applications. For BPU testing, where schedules are more difficult to accommodate, a three-week testing window is standard.

 

Figure 1. Testing process followed for each test pass.

Bug tracking

Issues (bugs) discovered during testing are created and tracked in Team Foundation Server (TFS), a Microsoft solution for Application Lifecycle Management. TFS provides the back end to Visual Studio and Microsoft Test Manager (MTM), a component of Visual Studio Ultimate and Visual Studio Premium used to help plan, manage, and execute the testing process.

Microsoft IT tracks bugs associated with the application, from the time the bug is filed by the tester through to resolution. Before testers can enter a test result of “fail” they must enter the bug number related to the issue, which promotes entering bugs into the test process for timely investigation. Bugs are triaged to determine proper handling:

·     Microsoft IT performs a rough triage to determine if the bugs qualify as something that must be relayed to internal product groups or handled by the BPU who owns the application. For example, fit and finish bugs, any minor UI issues—such as text or table alignment or graphic placement—are handled by the BPU not the product group.

·     A second level of triage is performed with the product groups to identify the correct severity/priority levels for the bug and places the bug in the queue for investigation by a product group developer or tester.

·     Towards the end of the test program, further triage is performed to determine which bugs will be addressed by the product group prior to release and which will not, requiring the application owner to change the application itself.

 

Test passes

During Windows 8 testing, Microsoft IT had three major test passes performed by the BPUs and some additional shallow testing (simple click-through testing with no workflow testing or data analysis) on non-primary applications performed by a dedicated team. With Windows 8.1, Microsoft IT changed their testing team model to include the embedded testing team who performed frequent, in-depth testing on primary applications. This resulted in:

·     A high number of meaningful bugs for the product groups and reduced the number of bugs requiring remediation per test pass.

·     Elimination of reversals where an application passing on the last test pass would be discovered as broken on the next pass due to thousands of lines of code changing between major BPU test passes.

·     Application compatibility testing that focuses on primary applications during each test cycle. Microsoft IT asks for voluntary participation from internal groups to identify these key applications that the business depends on, and Microsoft IT requires testing during every scheduled 5-week test cycle (1-week test cycles in old testing model). With voluntary participation and around-the-clock VM availability, Microsoft IT saw a significantly higher volume of applications tested per test cycle than in past programs.

 

With Windows 10 testing, Microsoft IT is performing 1-week test passes for each product tested during the 3-month test cycle. The embedded testing team performs 1-week test passes on 100 primary LOB applications, and the full Microsoft IT testing team including BPUs perform full testing on 200 critical LOB applications during a 3-week test window.

Product	Duration	Unique applications tested*
Windows Vista	2-3 week testing cycles	416
Windows 7 / Internet Explorer 8	1-week testing cycles	248
Internet Explorer 9	1-week testing cycles	313
Windows 8 / Internet Explorer 10	5-week testing cycles	344
Windows 8.1 / Internet Explorer 11		277
Windows 10 (estimated)	3-month testing cycles with weekly test passes	300

Table 2. Total number of unique applications (primary and secondary) tested per Windows deployment program.

*Although the application portfolio had grown significantly over the years, the number of unique applications tested did not grow over time as a result of Microsoft IT’s testing methodology.

LOB application testing during the Windows 8.1 testing program was synchronized with the product release milestones. Figure 2 shows product release milestones, build IDs with teams responsible for testing, and key test cycle dates.

Figure 2. Windows 8.1/Internet Explorer 11 test program schedule.

 

In the Windows 10 LOB application compatibility testing program, Microsoft IT carries out monthly cadences throughout the program with small dogfood deployments.

Figure 3. Windows 10 test program schedule.

Test matrix

A complete compatibility test matrix includes operating systems, platforms, and browser versions—and if required, productivity suite. Building a robust testing matrix to enable targeted testing remains the most important factor in reducing the impact and resource costs associated with application compatibility testing.

Microsoft IT limits their testing matrixes to reduce the number of variables to more easily identify the source of any bugs found. For example, a new OS is tested with the previous version of Office to limit bugs to the new OS and version of Internet Explorer. If both Windows and Office release at the same time, the new version of Windows is tested with the old version Office, and the new version of Office is tested with the old version of Windows. From there a judgment call is made at the point at which both technologies appear to be stable enough to streamline testing to only the new version of Windows with the new version of Office.

For Windows 8 application compatibly testing, the test matrix included two operating systems (Windows 8 and Windows 7 SP1 for Internet Explorer 10 legacy OS testing), one platform (PC-based Windows), three versions of Office (Office 2010, Office Professional 2013 and Office 365 ProPlus side-by-side), and two browser versions (desktop and modern browser). Microsoft IT added Modern Browser mid-release and included it as an optional, but highly encouraged testing configuration.

The Windows 8.1 application compatibility test matrix included two operating systems (Windows 8.1 and Windows 7 SP1 for Internet Explorer 11 legacy OS testing), two platforms (PC-based Windows and Windows RT), and two browser versions (desktop and modern browser).

For Windows 10 and Office 2016, the application test matrix includes testing of two operating systems, both Internet Explorer 11 and Microsoft Edge, and two versions of Office on PC and other device platforms.

Windows	Browser	Office
Windows 8.1	Internet Explorer 11 Desktop Browser	Office 2016
Windows 8.1	Internet Explorer 11 Modern Browser	Office 2016
Windows10	Internet Explorer 11	Office 2013
Windows10	Microsoft Edge	Office 2013
Windows10	Internet Explorer 11	Office 2016
Windows10	Microsoft Edge	Office 2016

Table 3. Windows 10 test matrix.

The Windows 8.1 LOB application compatibility testing program included PC and RT devices, which were the only devices officially supported by Microsoft IT. Moving forward, Microsoft IT is aligning their testing program with the Microsoft mobile first—cloud first strategy to test LOB applications on all devices and platforms, including competitive platforms. This helps ensure that not only Microsoft internal users but any user that interacts with Microsoft LOB tools—regardless of the platform—has a good experience.

Automation

Testing should be automated where possible. Automation provides for much quicker testing, resulting in more applications tested in the same period of time. Microsoft IT uses a combination of approaches to automate their test cases.

·     The virtual machine (VM) testing environment provides for an efficient, safe, and non-destructive way to test configurations before a wide-scale rollout without impacting the day-to-day hardware.

·     Scripting automates test cases wherever possible. Relying on manual testing for things that cannot be automated for whatever reason, for example with applications that experience frequent changes to the UI.

·     Visual Studio TFS allows a tester to record cursor clicks and data entry during testing and to “replay” those steps on subsequent testing cycles to automate the steps they followed in testing.

 

Microsoft IT is able to automate 50 to 85 percent of their test cases with the rest of the applications requiring manual testing according to scripts developed to minimize the level of effort involved.

 

Web application handling

Over the years Internet Explorer has evolved, improving performance, addressing security issues, and increasing compliance with current Internet standards. While these changes have greatly improved the product, they present some challenges to both Microsoft and Microsoft enterprise customers who need to weigh the benefits of a new modern browser against compatibility issues with legacy web-based LOB applications and websites.

With 93 percent of their LOB portfolio being web-based applications, Microsoft IT must ensure that compatibility issues do not hinder the business before deploying a new version of Internet Explorer internally. Microsoft IT works with the Internet Explorer product team to leverage and test successive iterations of new features to address long-term application compatibility goals for Internet Explorer. Testing started with the Release Preview for Windows 8.1 and Internet Explorer 11, and LOB application testing found:

·     Approximately 410 bugs were filed for the Internet Explorer team. Of those, about 49 percent of the bugs reported were resolved by adding the site to the Microsoft Compatibility View list, 2 percent of the bugs were fixed by changing Internet Explorer code, and the remainder were left for resolution by the app team. The Microsoft Compatibility View list is a list of websites with compatibility issues that is maintained by Microsoft and pushed to users worldwide. It originally put sites in Internet Explorer 7 document mode but was updated to put sites into any existing document mode.

Customers can now manage their own list of sites that render in any document mode, using the Enterprise Mode Site List. For more information on how the public Compatibility View List works, see Understanding the Compatibility View List. For more information on the Enterprise Mode Site List, see Fix web compatibility issues using document modes and the Enterprise Mode site list.

·     Sixty-eight applications that weren’t working were reported to the Internet Explorer team and were added to the Enterprise Mode Site List.

·     There were still many applications that didn't work in Internet Explorer 11 due to enabling higher security features such as Enhanced Protected Mode. Microsoft IT allowed Windows 7 users to opt out of the Internet Explorer 11 deployment to reduce the impact on Windows 7 legacy application users.

o  Approximately 10 percent of the 30,000 Windows 7 systems opted out of early Windows 8 deployment. The majority of these were due to a few third-party tools that weren't initially compatible with Internet Explorer 11.

o  Microsoft IT worked with those third-party tools companies to upgrade their tools to work with Internet Explorer 11, which unblocked the upgrade.

Enterprise Mode

As Microsoft IT continued their application compatibility testing, new Internet Explorer technologies were released. In April 2014, Enterprise Mode for Internet Explorer 11 was released. Enterprise Mode provides higher-fidelity Internet Explorer 8 emulation that works well with sites designed for older versions of Internet Explorer. In November 2014, functionality was improved; web paths added to the Enterprise Mode Site List can now be rendered either in Enterprise Mode or in any of the document modes in listed in Figure 4.

Figure 4. Enterprise Mode Site list structure.

Initially, three sites were placed on the Enterprise Mode site list. Two sites were eventually fixed and removed, but one site had an external dependency that was not resolvable and remains on the Enterprise Mode site list. To learn more about Enterprise Mode, see What is Enterprise Mode?

Moving to Internet Explorer 11 in Windows 10, Microsoft IT had another seven sites that worked in Internet Explorer 11 on Windows 8.1 but not the pre-release of Internet Explorer 11 on Windows 10. To resolve this, Microsoft IT placed the original sites added to the Microsoft Compatibility View list plus these seven new sites on the Enterprise Mode site list. Microsoft IT uses the Enterprise Mode Site List Manager tool to maintain their own list of sites that carry compatibility issues with Internet Explorer 11. The tool creates an XML file that Microsoft IT hosts. To learn more about the Enterprise Mode Site List Manager tool, see Enterprise Mode Site List Manager tool, and to download the tool, go to Enterprise Mode Site List Manager in the Microsoft Download Center.

Anytime Microsoft IT adds a site to the Microsoft Compatibility View list or to the Enterprise Mode site list, they work with the site owners on a plan to upgrade the site. Some sites are very large and may take years to upgrade and modernize. Of the 2,100 LOB applications in the portfolio, only about 700 are in active development, and only about 250 are tested in a given test program. All new LOB applications must meet current “Get Modern” standards, while revision of legacy LOB applications is performed on a case-by-case basis if the cost is justified, keeping with the Microsoft “Get Current” strategy. Upgrading to modern standards is still the right long-term goal, but leveraging Enterprise Mode can facilitate upgrades.

To learn more about managing Enterprise Mode, see Turn on Enterprise Mode and use a site list.

Enterprise Site Discovery

Microsoft IT, like many enterprise customers, didn’t have a complete list of the LOB applications hosted on the corporate network. Microsoft IT had, in the past, tried several solutions to determine which websites were in use internally. The Enterprise Site Discovery Toolkit enabled Microsoft IT to gather this information.

Enterprise Site Discovery, available on Internet Explorer 8 through Internet Explorer 11, provides a way to better understand how sites and web apps render in Internet Explorer. The toolkit contains scripts and sample queries to enable Enterprise Site Discovery and that show useful information, such as the sites visited, document modes used, and ActiveX controls used.

To learn more about Enterprise Site Discovery, see Collect data using Internet Explorer Site Discovery. To download the Enterprise Site Discovery Toolkit, go to Site Discovery Toolkit in the Microsoft Download Center.

Web App Modernization

The Internet Explorer LOB Application Development team created tools and processes to drive the LOB modernization status across the Microsoft IT application portfolio. Automated, repeatable testing processes helped to modernize existing applications and to design new LOB applications for use within the new Windows ecosystem. To accomplish this, Microsoft IT leveraged developer-created automated tests, debugging tools available in Internet Explorer 11, and a reporting structure. The output generated is provided to the application owner to assist them in understanding what their developers must do to modernize the application.

Microsoft IT LOB regularly tests web applications across 14 scenarios and assign a modernization level based on test results that range from Standards Level to Modern Level to Showcase Level. If an application does not meet the modernization level criteria, then it does not meet standards requirements and needs updating.

Communication

Preparing for a deployment can be a very busy time. By providing multiple channels of communication between itself and the various project stakeholders, Microsoft IT ensures prompt delivery of feedback, sharing of suggestions for features and functionality with the product teams, and delivery of the latest support information with users. Microsoft IT uses a combination of the following communication channels:

·     LOB testing aliases. Email is one of the primary modes of communication in the organization because of the wide geographic distribution of the testers. By creating email group aliases, Microsoft IT shares status messages and other communication directly with the people involved in the project. Email communications are timely and immediate; they communicate the current status and any information stakeholders need to know at that time.

·     https://pointers. Microsoft IT leverages an internal, moderated support forum, https://pointers, to allow dogfood users to report their issues and to receive answers on how to resolve the issues from subject matter experts (SMEs).

·     LOB testing website. Microsoft IT uses the LOB Windows SharePoint Services site to share information about the current testing status (duplicating the schedule information sent in email), the current application list for testing, known issues with the applications in testing, and any supporting documentation for the applications and testing process that may be pertinent. The website is an effective archive of past communications and current information.

·     Schedule. The testing schedule itself is an effective form of communication among all members of the project—from testers to stakeholders. Keeping the testing schedule updated and communicating any changes to the entire team sets expectations for all parties and lets them coordinate with their own internal test schedules.

·     Testing instructions. If there are specific scenarios to be tested in a particular test pass, Microsoft IT communicates instructions both during scheduling and in specific email communications at the beginning of the test pass.

·     Daily and final status. Sharing the status of the LOB testing on a daily basis helps to identify areas of concern in the process. If the daily status indicates that an application is failing the test pass, the test pass can be canceled for that application until the product team can improve the build stability. The final status is the core piece of the project postmortem; it is a chance to list all results, successes, and elements to be improved for the next round of testing.

 

Reporting

One of the major design criteria for an application portfolio management tool is its reporting ability. Reports on application testing status, application version in use, application team contacts, and related information can save large amounts of time and enable in-depth analysis of applications in the organization. Microsoft IT’s tool enables them to track the progress of an application as it moves through the test process, see comments from the tester, pass suggestions to the product team, and document the user experience that the tester encountered during testing.

Microsoft IT uses their portfolio management tool to generate reports during each test cycle, including status report for stakeholders, bug reports for BPUs and product groups, and the status of bug fixes.

Remediation plan

Part of application compatibility testing is understanding how you remediate applications that are determined to be incompatible with your latest OS ecosystem and productivity suite. As mentioned previously, for web-based applications, you need to determine how you will approach modernizing your portfolio. When it comes to third-party applications, you must evaluate the impact on the business in terms of support costs.

Features built into Internet Explorer can help overcome incompatibility issues, enabling applications to “works with” the latest version of Windows and Internet Explorer, but may not be supported by the vendor or require that you upgrade to a newer version to keep your support contract in place. During testing of early releases of Windows 10, Microsoft IT experienced this. Most applications worked with Windows 10 but may not be fully supported by the vendor on Windows 10.

Risk tolerance

Every enterprise evaluates risk in a different way. Some enterprises feel that unless they test every application in the portfolio and remediate every issue, the risk of upgrading to a newer version of Windows, Internet Explorer, or the Office suite is too high. While others want to embrace new technology and are prepared for an “acceptable level of risk” associated with moving forward.

For Microsoft IT, the business has identified the key business-critical applications that must be tested. That set of primary applications defines their level of risk tolerance. Of those applications, if one breaks, Microsoft IT must have some workaround. That's why Microsoft IT keeps track of alternate remediation options such as Enterprise Mode, allowing users to opt out of an upgrade for business reasons, and so on to ensure users are up-and-running with those primary applications.

Microsoft is no different than other enterprises, every effort is made to ensure LOB application compatibility by testing on the earliest product builds available. For those applications that cannot be upgraded or require too much time to upgrade, Microsoft IT works with the application owners to create a plan to eliminate the applications compatibility issues. That was the process Microsoft IT used and continues to use to move sites out of the Enterprise Mode site list.

Planning for Windows 10

Windows 10 includes Internet Explorer 11. Microsoft IT is in the process of testing application compatibility with Internet Explorer 11 on Windows 10. Microsoft IT performed full LOB application compatibility testing with early releases. The first release had a pass rate of 100 percent. The second release tested has a pass rate of 99 percent.

Windows 10 will also feature Microsoft Edge (previously known as “Project Spartan”), the first browser with “DO” in mind. It’s personal, productive, and responsive—but most importantly, it takes you beyond browsing to doing. Enterprise customers should standardize on Internet Explorer 11 for the fastest web path to Windows 10, but can choose to deploy Microsoft Edge and use the two browsers side-by-side; Microsoft Edge can switch to Internet Explorer 11 as needed to support legacy web sites.

Microsoft IT is following the same methodology for LOB application compatibility testing in the Windows 10 testing program. They are looking for the same types of bugs to help improve Windows 10 and to resolve issues with LOB applications. The most significant change is that Microsoft IT is testing more often as more releases are coming out, with monthly testing by the embedded testing team and also by the BPUs.

Microsoft IT has minimized their LOB application testing on desktop applications as desktop applications are slowly dwindling and is shifting their focus to web-based applications. At the time this paper was written, Microsoft IT tested 248 unique browser-based LOB applications for compatibility with Microsoft Edge.

Testing team	Number of test passes	Unique applications tested per test pass
Embedded testing team	4	92
Full BPU team	2	156

Table 4. Microsoft Edge LOB application test matrix.

Microsoft IT used the following process to when testing LOB application compatibility on Microsoft Edge.

Figure 5. Microsoft Edge LOB application compatibility testing process.

To learn more about how features of Internet Explorer 11 and Windows 10 make migration easier, see Making it easier for Enterprise customers to upgrade to Internet Explorer 11 — and Windows 10 and Project Spartan and the Windows 10 January Preview Build. To learn more about changes to the browser in Windows 10, see A break from the past: the birth of Microsoft's new web rendering engine and Updates from the “Project Spartan” Developer Workshop.

 

Benefits

One of the goals of the LOB application compatibility testing process is to identify issues early in product cycles. By identifying issues early in the process and providing clear feedback to the product groups, Microsoft IT can minimize the support impact on users and help the product groups create high-quality products. In addition, Microsoft IT derived the following benefits as a result of their LOB application compatibility testing.

The information provided in the preceding graphic reflects the benefits from Windows 7, Windows 8, and Window 8.1 testing programs. Microsoft IT expects to continue to add more critical LOB applications to their portfolio without increasing their overall testing budget using the current centralized testing model. Once the Microsoft IT portfolio includes all LOB applications, they will have the ability to extend testing as an IT service offering.

 

Microsoft IT derived some additional benefits as a result of application compatibility testing specific to Internet Explorer.

·     Reduced effort to administer and support upgrading Internet Explorer to the latest version.

o  Enterprise Mode for Internet Explorer 11 helped resolve legacy web application compatibility issues allowing Microsoft IT to deflect an average remediation cost of approximately $50,000 per application.

o  Enterprise Mode Site Manager made it easier to provide a granular level of compatibility.

o  Enterprise Site Discovery made it easier to gather information about how websites are being used.

·     Reduced helpdesk incident support rate. Enabling Enterprise Mode for Internet Explorer 11 allowed Microsoft IT to onboard a new acquisition from previous versions of Internet Explorer to Internet Explorer 11. Microsoft IT added 10 legacy websites to the Enterprise Mode Site list resulting in zero helpdesk calls for legacy applications.

Benefits of Internet Explorer features are highlighted in “The Total Economic Impact of Internet Explorer 11,” a commissioned study conducted by Forrester Consulting on behalf of Microsoft and published in March 2015. In the study, 47 percent of IT pros surveyed said the upgrade to Internet Explorer 11 was easier than expected—and 90 percent of these used backward compatibility features to ease their migration. Download The Total Economic Impact of Microsoft Internet Explorer 11 in any of 11 languages for more information.

Best Practices

The following are some of the best practices Microsoft IT wants to share from their experiences with LOB application compatibility testing.

·     Understand your environment. Survey your systems and identify LOB application dependencies with any OS, browser, or Office dependencies. Use:

o  System Center Configuration Manager to build your inventory of your environment to help with planning your deployment.

o  Microsoft Assessment and Planning (MAP) Toolkit to simplify the migration planning process.

o  Microsoft Application Compatibility Toolkit (ACT) to help determine your application compatibility with a new version of Windows.

o  Compatibility in Office 2013 to assess your Office 2013 compatibility.

·     Store historical data. Historical data helps you target your testing efforts on a subset of your portfolio, testing either the most complex applications, applications that historically have been known to have issues, or applications that are most critical for business continuance. Targeting the right subset of applications that leverage technologies in the same way enables testing efficiency and reduces the number of applications that you need to test.

·     Create a test plan. Your test plan and test matrix should be sufficient enough to enable you to test the various configurations in your environment and to represent the typical LOB application, minimizing the need to test every LOB application in your portfolio.

·     Gain efficiencies through automation. Use a test environment based on VM technology to perform a large number of tests in a short period without impacting your day-to-day hardware. Automate test cases wherever possible to further increase efficiency and to reduce test times.

·     Capture detailed bugs. The more fields you can put into your bug templates to capture details, the more you can automate your reporting and the more dynamic the reporting can be. More fields increase your efficiencies in target-based testing by capturing trending data.

 

·     Develop a remediation plan for your LOB applications. Enterprises need to build a remediation approach into the application compatibility test plan based on the enterprise’s acceptable level of risk.

o  Reach out to your top application partners and let them know you are moving to a new version of Windows (Windows 8.1 or Windows 10) to understand their schedule for supporting the application on your target platform.

o  Consider the potential cost to the business by not moving forward to a new release of Windows and Internet Explorer, both of which provide more robust security features. If security is an issue, determine if you are willing to take the risk of not remediating LOB applications that are blocking the upgrade. Or are you willing to invest to upgrade your blocking applications to take advantage of the latest technology?

o  Consider whether hardware requirements impact your ability to move forward with an upgrade of an incompatible application or an upgrade of your OS or productivity suite.

 

The following are some of the best practices Microsoft IT wants to share specific to browser compatibility.

·     Upgrade to the latest version of Internet Explorer. Internet Explorer 11 has increased performance, improved security, better backward compatibility, and support for the modern technologies like HTML5 and CSS3 that power today’s websites and services.

o  Upgrade to Internet Explorer 11 before January 12, 2016 to remain supported. After January 12, 2016 only the most recent version of Internet Explorer available for a supported operating system will receive technical support and security updates.

o  If you’re already piloting Internet Explorer 10, consider continuing your pilot with Internet Explorer 11. Enterprise Mode may streamline your browser pilot and simplify your browser and OS migration.

o  If you are considering upgrading Windows, start evaluating Internet Explorer 11 with Enterprise Mode. Migrating to Internet Explorer 11 can ease your upgrade to Windows 10 and the next generation of software, services, and devices.

·     Evaluate legacy web applications. Start by using Enterprise Site Discovery as a means of inventorying your websites, then determine how to address compatibility issues for those sites.

o  Upgrade applications to modern web standards for the best long-term solution.

o  Leverage browser backward compatibility as a cost-effective, if temporary, solution.

o  Create Enterprise Mode Site list to target legacy web application compatibility issues.

o  Target specific sites instead of using a one-size-fits-all solution to address compatibility issues.

 

Conclusion

Application compatibility testing is crucial for the successful deployment of new software in an enterprise environment. It is the final gate through which an application must pass before deployment to general availability at Microsoft. Microsoft IT will not authorize the deployment of any application or new technology for general use until it passes LOB testing and has mitigations in place for any outstanding compatibility issues.

Microsoft IT has developed a methodical approach to LOB application compatibility testing. Their VM environment, dedicated testers, open testing model, leveraging of compatibility features of Internet Explorer 11, and focusing their testing efforts on a representative subset of their LOB portfolio, has increased efficiency and optimized their application compatibility testing process. This foundation paves the way for testing the next version of Windows.

 

Resources

·     Windows Developer Center

·     Internet Explorer TechCenter

·     Internet Explorer Developer Center

·     What is Enterprise Mode

·     Modern.IE

·     Stay up to date with Internet Explorer on the MSDN IEBlog

 

Related Case Studies

Microsoft IT improves LOB application testing, ensuring readiness for Windows

Published May 2015

Transforming a Legacy LOB Application into a Microsoft IT Modern App

Published April 2014

Using modern apps to streamline and standardize business processes

Published June 2014

5 IT Strategies for App Creation and Delivery

Published June 2014

Aligning the IT Organization to Deliver Modern Apps

Published July 2014

Responsive Web Design for Enterprises

Published December 2014

Published October 2013

Building for the Modern Web - Leveraging IE11 Developer Tools to Build Modern LoB Applications

Published December 2013

 

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

www.microsoft.com

 

www.microsoft.com/ITShowcase

© 2015 Microsoft Corporation. All rights reserved. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.