May 2012

Volume 27 Number 05

ALM Rangers - Migrating from an On-Premises Team Foundation Server to Team Foundation Service Preview Using the TFS Integration Tools

May 2012

In this article, I demonstrate a migration from an on-premises Microsoft Team Foundation Server (TFS) 2010 to Team Foundation Service Preview, which uses the latest TFS Integration Tools. You can use these tools during the preview period, or at least until the hosted service offers a more direct attach/detach experience. The intent of this article is to walk you through some of the product’s key features, supplementing the technical information and guidance found in the product documentation.

The Microsoft Visual Studio ALM (Application Lifestyle Management) Rangers developed this demo application. If you’re not familiar with the ALM Rangers, we are a group of experts who promote collaboration between the Visual Studio product group, Microsoft Services and the Microsoft Most Valuable Professional (MVP) community by addressing missing functionality, removing adoption blockers and publishing best practices and guidance based on real-world experience.

Understand TFS and the TFS Integration Tools and When to Use Them

The TFS product group and the ALM Rangers developed the TFS Integration Tools to integrate TFS with third-party systems for migration and synchronization of source control and work item data between servers. These tools can also be used to copy data between TFSs.  

As emphasized in the Migration Guidance document, before deciding to use these tools, you must plan, test and carefully evaluate required maintenance resources and costs.

The product documentation is included with the TFS Integration Tools. You’ll find it at Start\All Programs\Microsoft Team Foundation Server Integration Tools\Documentation.

The product documentation cover these main areas:

• Migration guidance takes you through the features and main artifacts of the TFS Integration Tools, the underlying TFS Integration Platform architecture and the ALM Rangers value-adds.
• Proof-of-concept (POC) checklists are templates and sample documents that let you fast-track your projects and assist you with migration POC engagements.
• Practical hands-on labs guide you through the TFS Integration Tools.
• Configuration examples contain a wealth of configuration file examples from previous POCs.
• Tooling setup contains tooling-specific documentation.
• Common conflicts and resolution is a compiled help file that documents conflicts and their resolution.
• TFS Integration Platform SDK is a compiled help file that summarizes the SDK for custom adapter developers.
• Reference sites such as the TFS Integration Tools TOC post, the TFS Migration and Integration Solutions site and the Getting Started with IBM Rational to TFS Migrations post are handy for exploring and evaluating the product for your business.

Refer to my blog post Where does one start? … part 1 (documentation) for a more detailed introduction to the product documentation. The two quick reference posters and the check sheets will come in handy as you navigate the documentation.

The TFS Integration Tools help you move yet keep the integrity of version control (VC) data, work item tracking (WIT) data and the links in both. They also help you consolidate team projects and migrate to a different team project-process template. Synchronizations, which aren’t covered in this article, become important only when a migration isn’t feasible. The Microsoft dogfooding environment (VSTS Pioneer TFS dogfood server) is a typical synchronization example in which a complete switch to the latest version of TFS is impractical because of the huge impact on the work environment and significant work stoppage.

Download the Correct Version

The TFS Integration Tools are available from both CodePlex and Visual Studio Gallery. The version at CodePlex has the latest Beta bits, including unreleased adapters. The Visual Studio Gallery version has the current, fully supported release of the tooling. For more information on the support options and especially on how to make an assisted support call when you’re in trouble, see my blog post How do I call for support.

Plan the Migration

As you can see in Figure 1, the migration featured in this article moves an AFE_TFSIterationAutomation team project from an on-premises TFS 2010 to an AFE_Demo team project hosted by TFS Preview. The migration is from a Scrum-based team project to a Scrum-based project on the target environment, which means that no custom field data mapping rules should need to be created.

Figure 1  Migrating from on-premises to hosted environment

The TFS Integration Tools offer extensive field and value mapping features, but these quickly introduce complexity. When migrating between different process templates, you need to invest time to design and configure the necessary field and value mappings. Similar process templates (i.e., Scrum 1.0 and Scrum 2.0) will require few, if any, field mapping rules, whereas Scrum to Capability Maturity Model Integration (CMMI) will be much more involved.

Prepare the Prerequisites

Three main prerequisites are necessary before you can begin your migration.

Install the latest TFS Integration Tools

The recommended version for migrations for Team Foundation Service Preview is the March 2012 (or higher, when available) release. Note that you can upgrade a March 2011 installation with the March 2012 release.

To install the tools, you use the Microsoft Team Foundation Server Integration Tools Setup Wizard. In the Custom Setup page, you select the way you want features to be installed. Note that for the demonstration in this article, support for neither the TFS Integration Service nor the IBM Rational environment is selected. TFS Integration Service is useful for performing background migrations and synchronizations and for gathering statistics, none of which are required for the demonstration.

Install Visual Studio Team Explorer

For all versions of TFS you want to work with, you need to install Visual Studio Team Explorer (VSTE). For example, if you install VSTE 2010 and VSTE 11, the TFS Integration Tools 2008 adapters won’t show up in the Start\All Programs\Microsoft Team Foundation Server Integration Tools\TFS Integration tool. Visual Studio Team Explorer is included with Visual Studio 2010 and Visual Studio 11, which means that you can skip this step if you already have the IDEs installed.

Ensure necessary permissions

Once the product is installed, you need to ensure that you have the minimum access permissions on both the source and target environments. To have the demo migration succeed, the migration user must be a member of the Team Foundation Service Group and have the appropriate credentials.

To verify connectivity, you connect to both the source and target team project, using VSTE on the machine where the TFS Integration Tools are installed.

Configure the Migration

You start by configuring the source and target end-points, which is a fairly simple and intuitive task. The first step is to start the TFS Integration Tools administrative utility by selecting TFS Integration from Start\All Programs\Microsoft Team Foundation Server Integration Tools. To create a new session, in the TFS Integration dialog, from the left navigation pane, under Configuration, select Create New. In the Choose a template dialog, select the VersionControlAndWorkItemTracking configuration template so that version control and the work item tracking template as well as their respective links are moved, as shown in Figure 2.

Figure 2  Select the VC and WIT template

Now it’s time to configure the Left Source, which is the on-premises TFS 2010 that was shown in Figure 1. The configurations  are shown in Figure 3.

Figure 3  Configure the sessions

Notice the numbers 1 through 4 in Figure 3, which refer to other configuration tasks:

1. In Workflow type, One-way migration is selected because the intent is to move from on-premises to the Team Project hosted by TFS Preview.
2. For the source Version Control Session Provider, TFS 2010 Migration VC Provider is selected.
3. For the Work Item Tracking Session Provider, TFS 2010 Migration WIT Provider is selected.
4. These two errors indicate that the target sessions still need to be configured.

Next configure the Right Source, the TFS Preview environment (refer to Figure 1) is configured, as shown in Figure 4. During this process, you’re prompted for credentials if you didn’t check the Stay signed in check box in the LiveID authentication dialog when testing connectivity from Visual Studio 11 during the prerequisite preparation stage.

Figure 4  Configure target sessions

Leave the VC paths unchanged, at the default, as shown in Figure 5. Review the configuration guide included with the installation for the supported configuration options.

Figure 5  Default VC paths

Change the WIT Queries to [System.TeamProject] = @project and [System.WorkItemType] <> 'Sprint',  which returns all the WIT types except for Sprint (Figure 6).

Figure 6 WIT query returning all but Sprint WIT

To configure the more advanced features, such as field and value mapping, you have to roll up your sleeves, click the XML buttons or the XML tab and enter the world of the XML configuration file, as shown in Figure 7.

Figure 7 XML configuration file editing

To demonstrate the field and value mapping features, let’s  map the user accounts shown in Figure 8. The user Willy-Peter Schaub, on the source system, will be mapped to user A@Test.com on the target. Anyone else will be mapped to user X@Test.com.

Figure 8 Static user mapping

The next step is to edit the configuration XML file by selecting Edit XML for the WIT session. Some of the configuration changes need to be done manually.The final configuration SettingXml section is shown in Figure 9.

<SettingXml>
    <WITSessionCustomSetting>
       <Settings />
       <WorkItemTypes>
           <WorkItemType LeftWorkItemTypeName="Bug"
                         RightWorkItemTypeName="Bug" fieldMap="FieldMap" />
           <WorkItemType LeftWorkItemTypeName="Product Backlog Item"
                         RightWorkItemTypeName="Product Backlog Item"
                         fieldMap="FieldMap" />
           <WorkItemType LeftWorkItemTypeName="Task"
                         RightWorkItemTypeName="Task" fieldMap="FieldMap" />
       </WorkItemTypes>
       <FieldMaps>
           <FieldMap name="FieldMap">
                <MappedFields>
                   <MappedField LeftName="*" RightName="*"
                                MapFromSide="Left" valueMap="" />
                   <MappedField LeftName="System.Description"
                                RightName="" MapFromSide="Left"
                                valueMap="" />
                   <MappedField LeftName="Microsoft.VSTS.Common.DescriptionHtml"
                                RightName="System.Description"
                                MapFromSide="Left" valueMap="" />
                   <MappedField LeftName="System.AssignedTo"
                                RightName="System.AssignedTo"
                                MapFromSide="Left" valueMap="UserMap" />
                </MappedFields>
           </FieldMap>
        </FieldMaps>
        <ValueMaps>
            <ValueMap name="UserMap">
                <Value LeftValue="Willy-Peter Schaub"
                       RightValue="A@Test.com" />
                <Value LeftValue="*" RightValue="X@Test.com" />
            </ValueMap>
         </ValueMaps>
     </WITSessionCustomSetting>
  </SettingXml>

Figure 9  XML configuration file extract

The work item types, Bug, Product Backlog Item and Task, are configured to process. The migration will ignore all other types. All work item types reference a common field map, named FieldMap, which creates the mapping shown in Figure 10.

Figure 10  Custom field mapping

Here’s how we did the mapping:

• We started with a * -> * mapping. We migrated all fields from the left to the right unless there was an override.
• We defined a System.Description -> nothing mapping to suppress the System.Description field from the source. The non-HTML field doesn’t exist in the target using the latest Visual Studio Scrum 2.0 - Preview 3 process template.
• We defined a mapping from the Microsoft.VSTS.Common.DescriptionHtml source field to the System.Descriptiontarget field. System.Description is of type HTML in the latest Visual Studio Scrum 2.0 - Preview 3 process template.
• We concluded by mapping System.AssignedTo from left to right and using a custom valueMap named UserMap.
• In the UserMap, we defined two values to map from Willy-Peter Schaub to A@Test.com and everyone else (*) to X@Test.com.

We then closed the XML file and verified that the changes we made appeared correctly in the work item type session Customs Settings section, as shown in Figure 11.

Figure 11  Work item type session mapping customization

Finally, we saved the configuration to the database by clicking the Save to Database button.

Run the Migration

To begin the test run for this migration, in the TFS Version Control and Work Item Tracking with Links dialog, under Current Migration in the left navigation pane, select Start, as shown in Figure 12. The tool provides a variety of visual feedback while processing the analysis and migration steps of the configured sessions.

Figure 12  Starting the test run for the migration

The migration status form is shown in Figure 13. The general indicator (1) displays the migration status—in this case, Stopped—and the number of conflicts. Each session has its own detailed status. You can see that there’s an issue with the Version Control Session (2) and that the Work Item Tracking Session successfully migrated 233 WIT changes.

Figure 13  Progress indicators on the migration status form

At the bottom of the status form, the History tab has bar charts to indicate progress. A red bar, as shown in Figure 14, highlights a problem (conflict). Yellow bars indicate work in progress and green bars indicate completed migration tasks, as shown in Figure 15.

Figure 14  History tab error

Figure 15  History tab success

The Output tab (Figure 16) shows the detailed migration log for the current session, which is written to disk. This log is usually one of the artifacts requested when submitting a support request, and it’s a useful tool for learning about the migration process and debugging a migration.

Figure 16  Output tab on the status form, showing the verbose migration log

See Troubleshooting an unhappy migration world for more information on analyzing the status and logging information as part of a troubleshooting or validation exercise.

Resolve Conflicts

Resolving conflicts can be a frustrating exercise. It pays to invest time to understand the conflicts, identify patterns and proactively plan to avoid known conflicts in future migrations. Features such as MergeScope and Cloaking help you aggressively scope a migration to proactively avoid conflicts and other problems.

To start, click the Resolve link, which takes you to the list of conflicts, shown in Figure 17.

Figure 17  Conflict list

The VC conflict was detected during the migration. The cause was that parts of the same upgrade template file existed in both the source project and the new target project that was created. We left this conflict in the demo to illustrate how conflict resolution works. We could have used features such as cloaking to proactively avoid this conflict. To find out more about cloaking, go to TFS Integration Tools – What is the difference between cloaking and scoping branches?

To resolve the conflict, we have to find the relevant changesets in the source Team Project (Figure 18) and in the target Team Project (Figure 19).

Figure 18  Source Team Project changeset

Figure 19  Target Team Project changeset

You need to decide whether you want the source, the target or a merged version of the two. If you need the source or a merged version, you have to manually check out the file in the target workspace, modify its contents and then check it in.

After resolving the conflict, restart the migration session, which results in a much friendlier view (Figure 20).

Figure 20  Final migration state

You can always find addition information on conflict resolution by launching the TFS Integration Conflict and Resolution help.

Validate the Migration

When the TFS Integration Tool reports “completed successfully,” you need to verify that both the VC and WIT data migrated as expected.  For example, examining a random Product Backlog item, we find that all the information, including history, was migrated (Figure 21).

Figure 21  Sample WIT item

More important, when looking at the bugs from the source system (Figure 22) and the bugs migrated to the target system (Figure 23), notice that the custom field-mapping configuration worked as expected.

Figure 22  Bugs list on source system

Figure 23  Bugs list on target system

Conclusion

The quick demonstration featured in this article describes what the TFS Integration Tools guidance refers to as a pilot migration, during which you familiarize your team with the tooling, the source data, the migration requirements and conflict resolution. We recommend that you perform a number of these pilot migrations before you implement the tooling in production. Unexpected conflicts in production will likely impact and block development teams—not a scenario we wish anyone to experience.

Additional Resources

When experimenting with the TFS Integration Tools, you’ll likely want to create a number of test Team Projects, and although not needed, you might want to reinstall the TFS Integration Tools to start with a clean slate. Experimenting with migrations can result in orphaned resources, such as workspace mappings.

Here are some command examples you might find useful to clean up Team Projects and workspaces created by migrations:

• Deleting an orphaned test Team Project

Use the tfsdeleteproject command to delete all your test projects:

tfsdeleteproject /collection:https://demoonly.tfspreview.com/DefaultCollection AFE_Demo

tfsdeleteproject /collection:https://demoonly.tfspreview.com/DefaultCollection AFE_Demo1

tfsdeleteproject /collection:https://demoonly.tfspreview.com/DefaultCollection AFE_Demo2

• Finding workspaces created by the migration

Run the command tf workspaces and look for workspaces created by the Migration Toolkit (Figure 24). The TFS Integration Tools typically clean up the workspaces as the session completes, but it’s also good to clean up once everything is working as expected.

Figure 24  Migration workspaces

• Deleting orphaned workspaces

Use the tf workspace /delete command to remove the orphaned workspaces:

tf workspace /delete /collection:https://demoonly.tfspreview.com/DefaultCollection "WILLYS-EAGLE8b877b267b7e4641a36657663184c18aalmrangers.tfsprevie;Willy-Peter Schaub"

Here are few URLs you might find useful:

• TFS Integration Tools (Platform) table of contents for team blogs
• Team Foundation Server Migration and Integration Solutions page
• Supported product on Visual Studio Gallery (March 2012 Release)
• Latest beta (but unsupported) product on CodePlex

Willy-Peter Schaub is a Senior Program Manager with the Visual Studio ALM Rangers at the Microsoft Canada Development Center. Since the mid-1980s, he’s been striving for simplicity and maintainability in software engineering. His blog is at blogs.msdn.com/b/willy-peter_schaub, and you can also find him on Twitter www.twitter.com\wpschaub.

Thanks to the following technical experts for reviewing this article:   Patricia Wagner, Bill Essary, Bijan Javidi, Doug Neumann, Thomas Schissler, Adam Jagocki