Authors:

Jezz Santos (Microsoft)
Edward Bakker (LogicaCMG)
Rene Schrieken (LogicaCMG)

Contributors:

Mark Nichols (Microsoft)

Reviewers:

David Scruggs (Microsoft)
Attila Hajdrik (Microsoft)

October 2007

Applies To:

.NET Framework 2.0
Visual Studio 2005
Visual Studio SDK 4.0
Guidance Automation Extensions/Toolkit 1.2
Visual Studio 2005 Team Foundation Server 1.0

Summary: Welcome to part 2 of the short series on packaging and building software factories with team foundation build.

Software Factory solutions vary in structure today. Typically, they are composed of multiple project types like the ones identified in this series (e.g. VSX, GAT and DSL packages). In team build scenarios, these project types present unique challenges when automated build processes are applied to them.

Get the reference implementation.

Download the Reference Implementation for this article. You can use this as an example in your own organization. Please read the EULA before using this reference.

Contents

Introduction
Team Build with Software Factories
The Build Process
Automating the Team Build
Automating GAT Package Registration
Automating DSL Package Compilation
Automating Model Validation
Summary
About the Authors

Introduction

Welcome to part 2 of the short series on packaging and building software factories with team foundation build.

In the first part of the series ‘Packaging with Visual Studio 2005’ we saw what project types are commonly used in software factory solutions today (VSX, GAT and DSL), and how to create a combined installation package for deploying such a software factory using either standard VS setup projects and WIX setup projects. In this second part of the series we are going to look at how to automate the development and build of such software factory solutions, and their installers, in a team foundation build environment using Visual Studio Team Foundation Server (TFS).

In normal solution development where source code files represent the primary design-time artifacts, there exist source language compilers to verify whether source code is valid, and these tools compile that source into assemblies for packaging with installers. This compilation is invoked both on a developer’s machine and easily automated on a build server, in team build environments.

However, project types such as: GAT and DSL (common to software factory solutions), utilize ‘intermediate design artifacts’ which require explicit validation and explicit pre-compilation or transformation before other source artifacts and can be generated or compiled and then packaged.  In the case of GAT package projects, the intermediate design artifacts are the package XML manifest file, the recipes and the templates. The verification (and pre-compilation to some extent) is performed by package registration process (recipe). In the case of DSL package projects, the intermediate design artifacts are the DSL definition file (*.dsl) and the text templates (*.tt) files. The verification step is performed by the model validation engine, and the pre-compilation or transformation is performed by any text template transformations.

Invalid intermediate design-time artifacts (e.g. invalid domain models or recipes and templates) in GAT and DSL projects, should have the same meaning as invalid source code (whether its hand crafted or generated). If the intention of a build process is to ensure all checked-in artifacts are valid for building, then it follows that invalid intermediate artifacts in GAT and DSL package projects should carry the same consequence as invalid source code - i.e. it breaks the build.

In very small development environments, verifying, transforming and compiling a software factory solution, containing these types of projects, is typically individually micro-managed by those who develop the solution. They themselves control the development of the projects and enact the various individual micro-validation and micro-transformation processes to ensure that: (a) the design artifacts are valid, (b) all generated source artifacts are synchronized with their design artifacts, and (c) the packages compile before checking in code or compiling installer packages.

However, in larger development teams, with multiple developers, potentially all working on same design artifacts, these manual ‘micro-processes’ are difficult to synchronize reliably across multiple team members regularly and reliably. Therefore, centrally managed build processes are designed to ensure completeness and validity of all artifacts before creating and packaging regular releases.

Clearly then, to ensure all design-time artifacts are valid, the verification, transformation and pre-compilation steps required by GAT and DSL package project design artifacts should be automated by the team build environment. In this way, the build processes can quickly identify which design artifacts are invalid, ensure generated artifacts are is sync, and synchronize those artifacts (design and generated) with all members of the team.

Team Foundation Server (TFS) provides the ideal platform and build environment for defining, hosting, enforcing development policies and executing automated team builds for these types of solutions.

Team Build with Software Factories

In order to automate the build of a software factory solution that contains VSX, GAT and DSL package projects on a TFS build server today, several unexpected issues will be confronted. These are mostly due to differences between the build server environment and the developer environments for these package technologies at compile and debugging time. But also, there are issues of how to manage the pre-compilation and validation steps of intermediate design artifacts of specific to these types of solutions.

Most of the build environment issues are related to tools, build dependencies, hard coded paths to various SDK’s and toolkit resources, and configuration settings that are valid on a developer’s machine but not on a build server (with no pre-knowledge of these package project types). This is particularly prevalent in centrally hosted or strictly managed IT environments.

Most issues encountered are related to DSL development and extensibility components developed against the Visual Studio SDK.

In this article we focus only on team build issues with VSX, GAT and DSL packages specifically and exclusively. This article is not intended as a comprehensive guide to describing the general setup of TFS Team Builds.

Furthermore, this article does not address all issues that could be potentially encountered in setting up a build process for your particular software factory. However, it does aim to identify some of the key common issues and provide information and resolutions that will further help to resolve other classes of issues that may be encountered specific to your software factory solutions.

Team Build Environments

Ideally, as build managers, we would like to minimize any special compensations and configurations required on the build server just to build these types of projects For example, in the case of software factories, installing and configuring SDK’s and toolkits such as: the Visual Studio SDK, the GAX runtime and the DSL Tools. If this was an assumed practice, then all build servers would need to be installed with all SDK’s and toolkits (and all potential versions of them) expected for all current and future software development projects. This is simply not realistic in most software development organizations with managed shared development and build infrastructures. A number of these restrictions are enforced and carefully managed by enterprise infrastructure administrators and are uncompromising. They must be worked around for many development teams.

Therefore, it makes more sense to take the reliance off the particular build infrastructure, and create a specific build environment per solution. In this way, the specific solutions can target specific toolkit versions independently, provide all dependencies and resources, and have little to no impact on an existing shared development and build infrastructures.

The basic approach we take in this article is one that avoids the installation and configuration of any non-standard (out-off-the-box) components (e.g. additional SDK’s, toolkits, etc) specific to the project types contained within any given the solution. The rationale behind this is simply that we don’t assume that in all cases, in all projects, and all organizations that the development team will have full control over the build infrastructure, and would be able to tailor the build infrastructure to suit its own exclusive needs.

It is possible in certain projects or organizations that some of these restrictions may not be strictly enforced and the issues arising from them therefore not encountered. For example, in dedicated development projects where the development team has full control over the build environment, and evolve and maintain it as the development progresses.

Regardless, versioning is still an issue. Some development projects may still need to target multiple versions of their dependent SDK’s and toolkits, and it may not be possible or difficult to install these dependencies side-by-side in the build environment.

The content provided in this article will serve to gain an understanding of the various issues encountered for team build scenarios, their causes and resolutions, and provide information and practical guidance to understand the dependencies that new software factory technologies like the DSL Tools and GAT have for automating the build of software factory solutions today.

Sample Code

This article comes with a Reference Implementation (RI) sample demonstrating how to implement the required changes to an example software factory solution. In addition, an open source community project (www.codeplex.com/sfteambuild) has been created to provide additional resources to supplement the guidance given in this document.

More Information

The remaining sections of this article discuss how to create and customize an automated team build process that is capable of verifying and building software factory solutions (containing VSX, GAT and DSL project types), and their solution installers.

Information about creating and customizing automated builds using TFS can be found on MSDN:

·         Customizing Team Foundation Build

·         MSBUILD Overview

Best practices for using and customizing TFS can also be found in the patterns & practices Team Development with TFS Guide.

The Build Process

In team development scenarios, normally a centralized build process (i.e. ‘Daily Build’, Continuous Integration (CI) etc.) is designed and managed to ensure that regular development changes to solutions from multiple developers in the team are verified across the whole source code base, and then synchronized for next development iteration. For software factory solutions containing VSX, GAT and DSL project types this represents unique challenges not usually encountered in ordinary language developments such as with ordinary project types. This is primarily because VSX, GAT and DSL project types, as Visual Studio extensibility technologies, utilize generation, compilation, registration and design-time tools. Furthermore, GAT and DSL project types introduce intermediate development artifacts that require special ‘pre-processing’ before compiling their source code, and being packaging.

In a normal centralized regular build process, the following conceptual steps occur for each development iteration (omitting steps for: source control labeling, testing, distribution, notification etc):

On Development Machine:

1.     Developer(s) checks-out source artifacts from source repository

2.     Developer(s) adds/modifies source artifacts

3.     Developer(s) compiles source artifacts, and runs tests locally

4.     Developer(s) checks-in source artifacts

On Build Server:

5.     Build Process retrieves latest source artifacts from source repository

6.     Build Process compiles source artifacts

7.     Build Process packages solutions (installers)

8.     Build Process delivers solutions for installation and testing

The difference that compiling software factory package project types (containing GAT and dsl project types) presents in this process are primarily to do with the environment within which the compilation is executed, and the verification/transformation steps of those project types.

To the above process list the additional steps need to be added to support the project types commonly found in software factory solutions:

On the Development Machine:

2a. Developer(s) validates intermediate artifacts (i.e. model definitions, recipes etc)

2b. Developer(s) transforms intermediate artifacts to source code artifacts (i.e. using text templates)

And then:

3a. Developer(s) registers packages with runtime environment.

On the Build Server:

5a. Build Process validates intermediate artifacts (i.e. model definitions, recipes etc)

5b. Build Process transforms intermediate artifacts to source code artifacts (i.e. using text templates)

In the following sections we will explore the differences between the development machines and build server environments that are involved in implementing these additional steps, and introduce techniques to address the various issues faced. Then we will delve into options on how to implement the missing steps (5a-5b) for the various package project types.

Automating the Team Build

The following sections contain information about general infrastructure required to implement team builds on a build server, and the technical details on how to address these issues for software factory solution development.

Solution Structure

The compile and automated build process (for VSX, GAT and DSL project types) requires a number of support files and dependencies (i.e. tools, assemblies, configuration, include files etc) which need to be made available to both the build process on the build server, and at compile and debugging time on the developer’s machines.

The basic approach taken in this article is to provide all the files required to build the solution within the solution itself. These files would include various command line tools for performing operations on the projects in the solution with their supporting dependencies, configuration and include files.

These tools and supporting files would normally be installed and configured by specific versions of required SDK’s and toolkits for the solution (e.g. Visual Studio SDK, GAX/GAT). However, because we are avoiding installation of these toolkits to the build server, special steps have to be taken to provide these files and their configuration to the solution itself.

Since these SDK’s and toolkits are normally installed to a developer’s machine, we have access to all resources to put into our solution.

In order to create the necessary solution structure detailed in the sections below, the following process is enacted.

1.     Install the required SDK’s and toolkits to a clean development machine.

2.     Copy the required resources from the development machine into the solution structure

3.     Modify the resources to reference and operate from the solution structure, rather than from resources installed to the local machine.

The last point is critical here, since these resources need to operate on both the developers’ machine and on the build server. In some cases this is challenging and special workarounds are involved. The rest of the paper is dedicated to itemizing these workarounds.

Supporting Build Resources

For team build projects in TFS, a root project folder is created (‘MyTFSProject’) containing the solution (‘MySoftwareFactory’) and TFS team builds (‘TeamBuildTypes’).

The ‘TeamBuildTypes’ folder contains a sub folder called ‘MyFactoryTeamBuild’ containing the TFS project file (‘TFSBuild.proj’) which is created by the TFS ‘Project Creation Wizard’ (PCW).

The software factory solution folder (‘MySoftwareFactory’) contains the actual Visual Studio solution files and all the projects. In addition, this folder contains an additional folder (‘BuildExtension’) which will contain specific versions of the support files required for this solution.

For example:

The following tables identify the required files to populate the folders defined for this TFS project.

These supporting files are all obtained from installing the Visual Studio SDK (v.4.0) to a developer’s machine.

*Where %VSSDK% is by default: ‘%ProgramFiles%\Visual Studio 2005 SDK\2007.02’

*Where %GAC% is by default: ‘%WINDIR%\assembly’

Solution Dependencies

The various SDK’s and toolkits involved in authoring software factory solutions (e.g. DSL Toolkit, GAX/GAT etc.) have various additional assembly dependencies that are required at development and build time in order to build the solution. These dependencies would also include any 3rd party components, application blocks, class libraries, or frameworks etc. used by the solution. Common libraries for software factories may include: Enterprise Library (EntLib) or the GAX Extensions Library  (GEL) or Designer Integration Service (DIS) etc.

Typically all dependencies for these SDK’s and toolkits are installed to developer’s machines by the relevant SDK, toolkit installers.

To build in both the developer’s machine and on in the team build these dependencies are copied from a developer’s machine to the ‘ExternalDependencies’ folder in the software factory solution.

The following table lists the specific dependencies of GAX and the DSL Tools.

*Where %VSSDK% is by default: ‘%ProgramFiles%\Visual Studio 2005 SDK\2007.02’

This approach makes the assumption that theses assemblies have few or no references to other configuration (registry settings, configuration files, includes etc) installed by the toolkit on the build server (other than those in the ‘BuildExtension’ folder. Unfortunately, this is not entirely the case in practice. Later sections in this article detail those individual cases and provide the workarounds that enable this approach.

For the solution to use these dependencies, now in the solution, the following changes need to be made to the project files to reference these dependencies in the solution structure rather than on the local machine:

1.     Modify all references (HintPath) in all project files (*.csproj). For example:

Ensure that the path is relative to the location of the project in the solution structure.

    <Reference Include="Microsoft.VisualStudio.TextTemplating,
Version=8.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,
processorArchitecture=MSIL">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\..\MySoftwareFactory\ExternalAssemblies\Microsoft.VisualStudio.TextTemplating.dll</HintPath>
    </Reference>

2.     Add the following element to the ‘tfsbuild.proj’ file, to ensure the team build knows where to find our external assemblies.

<AdditionalReferencePath Include="$(SolutionRoot)\MySoftwareFactory\ExternalAssemblies" />

3.     Add the following <PropertyGroup> element containing the <SolutionDir> property to the ‘tfsbuild.proj’ file, to allow quick reference in other MSBUILD files to the software factory solution directory.

<PropertyGroup>
  <SolutionDir>$(SolutionRoot)\MySoftwareFactory\</SolutionDir>
</PropertyGroup> 

Developer or Team Build?

As we will see throughout the following sections in this document, some of the build steps that will need to execute will be either optional or different for a team build on the build server, than an individual build on a developer machine. To differentiate the build environments we have to implement a mechanism that detects in which environment the build is being executed. In other words, we need to know if we running a local build that is executed by the developer or a team build running on the build server.

In fact, there are 3 different build environments we need to consider:

·         Visual Studio Build – a build executed by a developer, on their own development machine inside the Visual Studio IDE

·         Team Build – a build executed by TFS (manually or scheduled), on the build.

·         Desktop Build – a build explicitly executed manually, on the development workstation using the command 'msbuild.exe tfsbuild.proj'.

A ‘DesktopBuild’ and a ‘TeamBuild’ are very similar in nature except that ‘DesktopBuild’ does not perform a ‘GetLatest’ function from source repository, will not ‘Label’ the source tree and will not determine the changeset.

When using MSBUILD tasks (as we will use primarily in following sections), one common way to achieve this is to use the ‘IsDesktopBuild’ and ‘BuildingSolutionFile’ properties as conditions to test in the tasks.

The ‘IsDesktopBuild’ property is declared in the ‘Microsoft.TeamFoundationBuild.targets’. The ‘BuildingSolutionFile’ property is declared and assigned automatically by MSBUILD.

The following table lists the values of each of these properties in each of the build environments.

One caveat with using the ‘IsDesktopBuild’ property is that it is not defined in many target files by default. This property will have an ‘empty’ value in a Visual Studio build, so we initialize it to a value of ‘true’ as the default value. Therefore we need to be explicitly define it in all MSBUILD target files where it will be tested.

We simply add the following <IsDesktopBuild> element to all target files where we need to differentiate between a build on the development machine and a build on the build server (within the first <PropertyGroup> section).

<IsDesktopBuild Condition="'$(IsDesktopBuild)' == ''">true</IsDesktopBuild>

MSBUILD Tasks

MSBUILD tasks provide a managed, maintainable and reusable way to implement many of the tasks we wish to perform on both the development and build server machines during the build process. The MS Build engine uses a ‘target’ to group several ordered build tasks together. A target often represents a logical unit of work like: deleting files from a directory or compiling source code files, etc.

Normally, targets are declared in the project files that use them, but for reuse and maintainability purposes these targets are often defined in shared files that are imported into the project files. For example:

<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />

For a list of MSBUILD properties that can be used as paths for theses includes see ‘MSBuild Reserved Properties’

In case of software factory solutions (that include VSX and DSL’s packages) there are commonly three target files (part of VS SDK installation) that we have to deal with:

·         Microsoft.DslTools.targets – contains tasks specific to DSL solutions (%ProgramFiles%\MSBuild\Microsoft\VisualStudio\DSLTools\v2.0)

·         Microsoft.VsSDK.targets – contains tasks for normal VSX packages (%ProgramFiles%\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Tools\Build)

·         Wix.targets – contains tasks for WIX setup projects (%ProgramFiles%\MSBuild\Microsoft\VisualStudio\DSLTools\v2.0)

Currently, GAT project types do not make use of additional MSBUILD tasks, and therefore there are no specific target files for these project types.

These targets contain a number of tasks that have multiple dependencies on files, tools and registry settings normally installed by the various SDK’s and toolkits on developer machines.

To use these tasks in a team build scenario (where we want to minimize the impact of installing toolkits on the build server), we need to modify not only the project files that reference the targets, but also the targets files themselves (provided by the various toolkits) and enhance them. We need to do this because the target files that are installed by default won’t be using the ‘IsDesktopBuild’ condition – as noted before. We need to provide our own versions of these target files in our source tree, and we need to redirect or provide the location of dependencies of the targets to dependencies and support files in our source tree. We don’t want to have to update the target files on every developer’s machine, so instead use a single modified target file from within the source code structure to be accessible to the solution no matter where it is built.

Modifying Target Files

For each of the targets files referenced above, copy the original version from a development machine and add to the ‘BuildExtension’ folder of the solution.

For more details on creating and customizing build tasks see: Walkthrough: Customizing Team Foundation Build with a Custom Task on MSDN.

Referencing Targets Files

In all project files that use any of the above targets, we need to make sure we import the modified target files that are available on the build server.

In all project files of the software factory solution (*.csproj or *.vbproj), replace all occurrences of <Import> statements, such as:

<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\DSLTools\v2.0\Microsoft.DSLTools.targets" />

With the appropriate <Import> statements:

<Import Project="$(SolutionDir)MySoftwareFactory\BuildExtension\Microsoft.DSLTools.targets" />

Notice that throughout this article, the examples paths given may be different depending on the naming used and where the solution is located in respect to the current project in the source tree.

VSX Project Specific Issues

The following issues are specific only to any VSX package projects (includes DSL package projects) contained within the software factory solution.

VSSDK Tools Dependencies

In the Microsoft.VsSDK.targets file, the ‘FindSDKInstallation’ target has a dependency on the ‘FindVsSDKInstallation’ task.

<Target Name="FindSDKInstallation"
          Condition="'$(VsSDKInstall)'==''" >
    <FindVsSDKInstallation SDKVersion="$(VsSDKVersion)">
      <Output TaskParameter="InstallationPath" PropertyName="VsSDKInstall"/>
      <Output TaskParameter="IncludesPath" PropertyName="VsSDKIncludes"/>
      <Output TaskParameter="ToolsPath" PropertyName="VsSDKToolsPath"/>
    </FindVsSDKInstallation>
</Target> 

The ‘FindVsSDKInstallation’ task uses the ‘Microsoft.VsSDK.Build.Tasks.dll’ assembly (from the VS SDK) to locate some values from the registry and populate some output parameters (i.e. VsSDKInstall, VsSDKIncludes and VsSDKToolsPath).

<UsingTask TaskName="FindVsSDKInstallation" AssemblyFile="Microsoft.VsSDK.Build.Tasks.dll" />

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

Instead, we can provide the values of these properties by defining them in the target file to avoid the registry lookup, by adding the following properties.

<PropertyGroup>
    <VsSDKInstall>$(SolutionDir)MySoftwareFactory\BuildExtension\</VsSDKInstall>
    <VsSDKIncludes>$(SolutionDir)MySoftwareFactory\BuildExtension\inc\</VsSDKIncludes>
    <VsSDKToolsPath>$(SolutionDir)MySoftwareFactory\BuildExtension\</VsSDKToolsPath>
</PropertyGroup>

Where the paths above are relative to the solution files in the source tree structure.

Registering VSX Packages

For debugging purposes, VSX packages are automatically registered with Visual Studio when rebuilt by executing the following command line:

devenv.exe /setup 

This re-registers the VSX package with new instances of Visual Studio by refreshing a number of its caches. This is implemented with a number of targets in the Microsoft.VsSDK.targets file, such as: ‘SettingUpDevenv’, ‘GenerateParserCodeFromGrammar’ and ‘GenerateCodeFromLex’.

However, this registration is not required as part of an automated build process, since the build process does not debug the compiled packages. These targets therefore should not be executed on the build server, and requires that the ‘IsDesktopBuild’ condition restrict their execution to development machines only.

  <Target Name="SettingUpDevenv"
          Inputs="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')" 
          Outputs="$(DevenvSetupStateFile)" 
          DependsOnTargets="$(SettingUpDevenvDependsOn)"
	  Condition="'$(IsDesktopBuild)'=='true' >
    <DevEnvSetup CTOFiles="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')"
                 ProductVersion="$(ProductVersion)"
                 RegistryRoot="$(TargetRegistryRoot)"
                 StateFile="$(DevenvSetupStateFile)" />
  </Target>

CTC Task Dependencies

CTC command structure is used by the DSL Tools for versions contained in VSSDK 4.0 and below. For higher versions of the DSL Tools, a new type of command structure is used.

The ‘CTCCompile’ task in the Microsoft.VsSDK.targets file depends on a registry key that holds the location of the VS SDK resources. In previous sections we have avoided the registry lookup by providing hardcoded property values. However in the case of the ‘CTCCompile’task this workaround is not effective due to architecture of the ‘CTCCompile’ task.

In this case we can’t prevent a registry key lookup on the build server. Instead, we employ a strategy to set the registry key using a new target called ‘SetVsSDKInstallDir’. In this way provide the required value, and again avoid the installation of the VSSDK.

By adding the ‘SetVsSDKInstallDir’ target we introduced a dependency on the ‘Microsoft.Sdc.Common.tasks.dll’ assembly from http://www.codeplex.com/SdcTasks. This Microsoft provided library contains a number of reusable MSBUILD targets for creating customized build processes.

<Target Name="SetVsSDKInstallDir" Condition="'$(IsDesktopBuild)'!='true'">
    <Registry.CreateKey RegistryHive="LocalMachine" Key="Software\Microsoft\VisualStudio\VSIP\$(VsSDKVersion)" />
    <Registry.Set RegistryHive="LocalMachine" Key="Software\Microsoft\VisualStudio\VSIP\$(VsSDKVersion)" DataType="String"  Value="InstallDir" Data="$(SolutionDir)MySoftwareFactory\BuildExtension" />
</Target>

Then we modify the ‘CTCCompile’ target to make it dependent on our new target. This ensures that the ‘SetVsSDKInstallDir’ target is executed before the ‘CTCCompile’ target.

The ‘CtcCompile’ target also has a dependency on the C++ compiler/linker which is also required to be installed on the build server. (See next section.)

<Target Name="CtcCompile"
          Inputs="@(CtcFile)" 
          Outputs="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')" 
          DependsOnTargets="$(CtcCompileDependsOn);SetVsSDKInstallDir" 
          Condition="'$(BuildingProject)'!='false'">
    <Ctc IncludeDirs="@(CtcIncludePath);$(VsSDKIncludes);$(VsSDKIncludes)\office10" 
         Inputs="@(CtcFile)" 
         OutputFile="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')"
         ProductVersion="$(ProductVersion)" 
         SDKVersion="$(VsSDKVersion)" >
    </Ctc>

Since the ‘SetVsSDKInstallDir’ task creates a registry key that is not expected to already exist on the build server, due to the fact the VS SDK is not installed, it may be prudent to delete the registry key after the task has executed, leaving no footprint of this task having been executed.

CTC Compiler

VSX packages utilize the CTC compiler for compiling *.ctc resource declarations. The CTC compiler depends on the C++ compiler/linker (CL.exe). The ‘CTCCompile’ target used for VSX packages also depends on the CTC compiler.

In order to build VSX package projects correctly, the Visual C++ language component of Visual Studio is required to be installed to the build server. This is unavoidable.

One means to install the compiler is by running the Visual Studio installer on the build server, and selecting only the ‘C++ Language’ component.

Regardless of how you install the C++ compiler, you need to ensure that the path to the compiler is defined in the machine’s PATH environment variable on the build server, for example:

PATH = %ProgramFiles%\Microsoft Visual Studio 8\VC\bin;%ProgramFiles%\Microsoft Visual Studio 8\Common7\IDE

DSL Project Specific Issues

The following issues are specific only to any DSL package projects contained within the software factory solution.

DSL Tools Dependencies

The ‘Microsoft.DslTools.targets’ file contains a target that provides the location where the DSL Tools dependency assemblies are located.

<UsingTask TaskName="FindDslToolsInstallation" AssemblyName="Microsoft.VisualStudio.Modeling.Sdk.Utilities,
Version=8.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
      <PropertyGroup>
      <CtcCompileDependsOn>$(CtcCompileDependsOn);DSLToolsAppendIncludePath</CtcCompileDependsOn>
      <MergeCtoResourceDependsOn>$(MergeCtoResourceDependsOn);DSLToolsCreateCtcResources</MergeCtoResourceDependsOn>
      </PropertyGroup>
      <Target Name="DSLToolsFindInstallation"
          Condition="'$(DslToolsIncludes)'==''">
            <FindDslToolsInstallation>
                  <Output TaskParameter="DslToolsIncludes" PropertyName="DslToolsIncludes" />
            </FindDslToolsInstallation>
      </Target>

The ‘FindDslToolsInstallation’ target uses the assembly ‘Microsoft.VisualStudio.Modeling.Sdk.Utilities’ to locate the DSL tools dependencies by looking up the value in the registry.

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

Instead, we can provide the values of these properties by defining them in the target file to avoid the registry lookup, by adding the following properties.

<PropertyGroup>
  <DslToolsIncludes>$(SolutionDir)MySoftwareFactory\BuildExtension\</DslToolsIncludes>
</PropertyGroup>

Adding Files to GAC Tasks

For debugging purposes, DSL package assemblies are automatically added to the GAC when rebuilt. This is implemented in the target file (Microsoft.DslTools.targets) containing the following 2 tasks: ‘DSLToolsGacInstall’, and ‘DSLToolsGacClean’.

However, adding assemblies to the GAC is not required as part of an automated build process, since the build process does not debug the compiled packages. These targets therefore should not be executed on the build server, and requires that the ‘IsDesktopBuild’ condition restrict their execution to development machines only.

<Target Name="DSLToolsGacInstall"
DependsOnTargets="GetFrameworkPaths;Compile"
Condition="'$(GacTargetOutput)'=='true'
And '$(RegisterWithCodebase)'!='true' And '$(IsDesktopBuild)'=='true'">
<DslGacutil ToolPath="$(FrameworkSDKDir)bin"Assembly="$(TargetPath)" Uninstall="false" />
</Target>

General Project Related Issues

These issues apply to any projects contained in the software factory solution.

Conditional Pre/Post-Build Events

In any software factory solution there may exist pre-build or post-build events that are only appropriate to be executed on development machines, and not to be executed on build servers.

In the same way we control MSBUILD events in the project and target files, we can control the pre-build and post-build events by restricting their execution conditional on the ‘BuildingSolutionFile’ property.

<PostBuildEvent Condition="'$(BuildingSolutionFile)'=='false'">some commands</PostBuildEvent>

Setup Project Issues

The common types of setup projects available for building factories include: standard VS setup projects, and WIX setup projects. Standard VS setup projects and WIX 2.0 setup projects cannot be compiled as part of an automated build in TFS because neither of these project types are MSBUILD compatible, and at solution compile time these projects will be ignored (with warnings). However WIX 3.0 setup projects are MSBUILD compatible and come with their own MSBUILD targets, but WIX 3.0 is not recommended because of its release status, and incompatibility with the WIX 2.0 projects and targets currently used by the DSL Tools.

At the time of writing, WIX 3.0 is still under construction, and considered ‘unstable’.

In the first part of this series ‘Packaging with Visual Studio 2005’ we described in detail a method of creating WIX 2.0 compliant setup projects that are MSBUILD compatible. These types of projects are built upon standard project types (Class Library), and use MSBUILD targets from the WIX 2.0 toolset. However, if you wish to compile these setup projects as part of an automated build process using TFS, you need to make some modifications in order to compile them as part of the build processes described in this article.

The Reference Implementation (RI) included in this part of the series builds upon and extends the RI from the first part of this series. It adds all the resources and dependencies, and modifies the MSBUILD targets required. In order to ‘upgrade’ or ‘teambuild-enable’ the setup projects that are used, a number of changes are required to the setup project files for each setup project type.

VSX Setup Projects

The following changes are specific only to any VSX package setup projects contained within the software factory solution.

The following changes are made to the setup project file (*.csproj).

1.     Add the following properties to the first <PropertyGroup> element.

    <SolutionDir Condition="'$(SolutionDir)'==''">..\..</SolutionDir>
    <OutDir Condition="'$(OutDir)'=='' and '$(BuildingSolutionFile)'=='false'">$(MSBuildProjectDirectory)</OutDir>
    <ToolPath>$(SolutionDir)BuildExtension\WIX\v2.0</ToolPath>
    <BaseInputPath>$(OutDir)</BaseInputPath>

Where the value of <SolutionDir> is a relative path from project file to solution, which may vary in your specific solution.

2.     Replace the current <Import> element with the following:

  <!--<Import Project="$(MSBuildExtensionsPath)\WIX\v2.0\wix.targets" />-->
  <Import Project="$(SolutionDir)\BuildExtension\WIX\v2.0\wix.targets" />

 GAT Setup Projects

The following changes are specific only to any GAT package setup projects contained within the software factory solution.

The following changes are made to the setup project file (*.csproj).

1.     Add the following properties to the first <PropertyGroup> element.

    <SolutionDir Condition="'$(SolutionDir)'==''">..\..</SolutionDir>
    <OutDir Condition="'$(OutDir)'=='' and '$(BuildingSolutionFile)'=='false'">$(MSBuildProjectDirectory)</OutDir>
    <ToolPath>$(SolutionDir)BuildExtension\WIX\v2.0</ToolPath>
    <BaseInputPath>$(OutDir)</BaseInputPath>

Where the value of <SolutionDir> is a relative path from project file to solution, which may vary in your specific solution.

2.     Replace the current <Import> element with the following:

   <!--<Import Project="$(MSBuildExtensionsPath)\WIX\v2.0\wix.targets" />-->
  <Import Project="$(SolutionDir)\BuildExtension\WIX\v2.0\wix.targets" />

DSL Setup Projects

The following changes are specific only to any DSL package setup projects contained within the software factory solution.

The WIX files (*.wxs) generated by the DSL setup project file assume that certain files to be packaged are located in a target subfolders ‘bin\$(Configuration)’, where the build configuration is typically either ‘Debug’ or ‘Release’ or other custom build configuration.

In team build, all compiled assemblies are sent to the ‘$(OutDir)’ folder, and the directory structure is quite different than that on the development machines.

One means to solve this issue for team builds is to replace the references for these files in the WIX files (*.wxs) to point to the ‘$(OutDir)’ folder on the build server.

Another approach might have been to change the text template (*.tt) files to simply generate the correct path locations in the generated WIX files (*.wxs) during a team build. However, that may be affected by new releases of the DSL Tools if those text template files may change.

Instead, a better approach simply ‘patches’ the generated WIX files (*.wxs) after their generation. That gives us also the possibility to make sure we have the file layout in our folder structure match the file layout in the WIX files (*.wxs) files.

The following logical steps are executed:

1.     Files are batched together:

a.     Setup supporting files from the ‘\Files’ project directory (i.e. eula.rtf and  readme.htm)

b.     Generated WIX files (*.wxs) from the main project directory

c.     Supporting files from the ‘\ExternalAssemblies’ folder in the source tree

2.     Files are Copied using the <Copy> task to the $(OutDir) directory

3.     WIX files (*.wxs) are ‘patched’ using the <Replace> task.

To implement this, the following changes are made to the setup project file (*.csproj).

1.     Add the following properties to the first <PropertyGroup> element.

    <SolutionDir Condition="'$(SolutionDir)'==''">..\..</SolutionDir>
    <OutDir Condition="'$(OutDir)'=='' and '$(BuildingSolutionFile)'=='false'">$(MSBuildProjectDirectory)</OutDir>
    <WixToolsPath>$(SolutionDir)BuildExtension\Dsl</WixToolsPath>
    <DslToolsRedistPath>$(SolutionDir)BuildExtension\Dsl\</DslToolsRedistPath>
    <BaseInputPath>$(OutDir)</BaseInputPath>

Where the value of <SolutionDir> is a relative path from project file to solution, which may vary in your specific solution.

2.     Replace the current <Import> element with the following:

  <!--<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\DSLTools\v2.0\wix.targets" />-->
  <Import Project="$(SolutionDir)\BuildExtension\dsl\wix.targets" />

3.     Add the following <Import> element.

  <Import Project="$(SolutionDir)\BuildExtension\Microsoft.DSLTools.targets" />

DSL Tools Dependencies

The DSL Tools use a modified version of the Wix.targets file. This targets file contains a target that provides the location where the DSL tools and the WIX tools.

<Target Name="GetToolLocations"
   Condition="'$(WixToolPath)'=='' And '$(DslToolsRedistPath)'==''">
   <FindDslToolsInstallation>
      <Output TaskParameter="RedistInstallerPath" PropertyName="DslToolsRedistPath"/>
      <Output TaskParameter="WixToolsPath" PropertyName="WixToolsPath"/>
   </FindDslToolsInstallation>
</Target>

The ‘FindDslToolsInstallation’ target uses the assembly ‘Microsoft.VisualStudio.Modeling.Sdk.Utilities’ to locate the DSL tools dependencies by looking up the value in the registry.

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

Instead, we provide the values of these properties by defining them in the project file to avoid the registry lookup. (See the previous section for property declaration)

All Setup Project Types

The following changes are specific to all package setup projects contained within the software factory solution, where they have any WIX files (*.wxs).

WIX files reference resource files (i.e. files and binaries etc.) using paths that are relative to the files on disk, primarily copied to ‘bin\Debug’ (depending on the current configuration). This works fine for Visual Studio builds, but not for team builds on the build server. In team build the projects don’t use separate project output folders, instead they have a single output folder.

To work around this, we need to replace the ‘src’ attribute for these resources and redirect the path on the build server to match the output path used in team build.

To replace the ‘src’ attributes we need to execute a custom target that makes this replacement.

1.     Add the following elements after the <Import> element in the project file.

<CreateItem> is used here because not all files are available when a normal <ItemGroup> would be evaluated.

  <PropertyGroup>
    <BuildDependsOn>BeforeBuild;$(BuildDependsOn);AfterBuild</BuildDependsOn>
  </PropertyGroup>
 
  <UsingTask TaskName="Replace" AssemblyFile="$(SolutionDir)\BuildExtension\MyCompany.FactoryTeamBuildUtilities.dll" />
  <Target Name="BeforeBuild" Condition="'$(BuildingSolutionFile)'=='true'">
    <CreateItem Include="Files\*">
      <Output TaskParameter="Include" ItemName="MoveFiles" />
    </CreateItem>
    <CreateItem Include="$(OutDir)\GeneratedCode\*">
      <Output TaskParameter="Include" ItemName="MoveFiles" />
    </CreateItem>
 
    <!-- we are going to change the *.wxs files, so keep them save -->
    <CreateItem Include="%(Compile.FullPath)" AdditionalMetadata="Original=%(Compile.FullPath)">
      <Output TaskParameter="Include" ItemName="KeepOriginal" />
    </CreateItem>
    <Copy SourceFiles="@(Compile)" DestinationFolder="obj\tempstore" />
    <Copy SourceFiles="@(MoveFiles)" DestinationFolder="$(OutDir)" />
    <Replace File="%(Compile.FullPath)" />
    <OnError ExecuteTargets="AfterBuild" />
  </Target>
 
  <Target Name="AfterBuild" Condition="'$(BuildingSolutionFile)'=='true'">
    <!-- let's get the original wxs files back -->
    <Copy SourceFiles="obj\tempstore\%(KeepOriginal.Filename)%(KeepOriginal.Extension)" DestinationFiles="%(KeepOriginal.Original)" />
    <!-- <RemoveDir Directories="obj\tempstore\" /> -->
  </Target>

The reason that this task is embedded in the project file, instead of in a shared targets file, is that each project it might need customizing to suit the requirements of the specific project (i.e. folder structure etc.) and this approach enables that flexibility.

The <Replace> task depends on the ‘Replace’ method in the ‘MyCompany.FactoryTeamBuildUtilities’ assembly, which is also referenced in the task. The ‘Replace’ function is implemented in the ‘MyCompany.FactoryTeamBuildUtilities’ assembly as follows:

using System;
using System.IO;
using System.Xml;
 
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
 
namespace MyCompany.FactoryTeamBuildUtilities.CustomBuildTasks
{
    public class Replace : Task
    {
        private string _file;
 
        [Required]
        public string File
        {
            get { return _file; }
            set { _file = value; }
        }
        public override bool Execute()
        {
            FileInfo fi = new FileInfo(this.File);
            fi.Attributes = fi.Attributes & ~FileAttributes.ReadOnly;
 
            XmlDocument wxsDoc = new XmlDocument();
 
            wxsDoc.Load(this.File);
            // select every attribute that is named 'src'
            XmlNodeList fileElements = wxsDoc.SelectNodes(@"//@src");
            foreach (XmlAttribute srcAttribute in fileElements)
            {
                // now replace the path, with a Replace Regex
                // keep the first part of the string, 
                // get rid of bin\Debug\ (if it exists) 
                // or get rid of the path
                // keep the rest of the path and/or file
                const string pattern = @"(?<root>.*?\\)(.*(bin\\(.*?\\){1}?)|.*\\(?=.*))(?<remainingpath>.*)";
                const string format = "${root}${remainingpath}";
 
                srcAttribute.Value = Regex.Replace(srcAttribute.Value, pattern, format);
            }
            wxsDoc.Save(this.File);
 
            return true;
        }
    }
}

It is also possible to implement the custom ‘Replace’ task as a custom XML task from www.codeplex.com/sdctasks, and avoid the extra ‘MyCompany.FactoryTeamBuildUtilities’ assembly. An XML task implementation of the replace function can be found at http://www.codeplex.com/sfteambuild.

Automating GAT Package Registration

GAT packages are defined in an XML manifest file. This file is parsed at registration time (to create runtime registration information), and also at runtime by GAX (to provide integration to Visual Studio).

If we want to ensure that a GAT package has been developed correctly at development time, this manifest file needs to be verified, as well as verifying all the types it refers to at build time.

Currently, this verification step is invoked during debug time manually by the developer when the package is registered. Ideally, this verification step needs to automated as part of the team build process to ensure GAT development artifacts are correct, as GAT developers are able to change the manifest file without having to re-register the package manifest.

Currently, there exists no automatable means to execute package GAT verification on the build server.

The Software Factories Toolkit from Clarius provides a simple means to execute this verification step manually, again at debugging time, but this cannot be automated on the build server.

Automating DSL Package Compilation

DSL package projects define a domain specific language (DSL) using a DSL definition file. This XML file is transformed (using numerous text templates within the DSL projects and DSL setup projects) into source code artifacts. Even minor changes to the DSL definition may require the re-transformation and generation of all source artifacts in these projects.

The DSL toolkit provides the ‘Transform Templates’ command in Visual Studio to perform this task manually. The DSL Tools also provide the TextTranform.exe command line tool to automate this task.

If we want to ensure that a DSL definition is correct at build time, the definition file first needs to be transformed, and all generated source artifacts compiled into the assemblies (and those assemblies packaged into the setup projects).

Currently, this transformation step is invoked during debug time manually by the developer when the DSL definition is changed. Ideally, this transformation step needs to be automated as part of the team build process to ensure the DSL definition is correct, as DSL developers are able to change the definition file without having to transform the templates.

Automating Text Transformation

The DSL tools provide the TextTransform.exe command line tool to transform text templates, and this tool can be automated on the build server.

There are a number of means to automate this transformation process using differing techniques and mechanisms as part of the build process (i.e. post-build events or MSBUILD tasks).

One of the key issues with automated transformation from intermediate design artifacts (e.g. model files) to source code is verifying that the source code represents the design artifacts. It may be almost impossible to verify by looking at the source code, since this would involve reverse engineering techniques.

In the case of the DSL Tools, the automated build process need to ensure that the generated source code represents the current state of the DSL definition. This is not possible looking at the source code, and so automatic transformation of the text templates needs to be executed to ensure that the source code represents the current DSL definition.

With any system when source files (hand crafted or generated) are source controlled, this automation presents some challenges, especially when automated on a central build server. The key issue here is that these generated source code files can be checked out and locked by any member of the development team. Since the automated build process would also need to check out these generated files to complete the transformation process, contention for file locks is a real issue. Failing to be able to check out a file on the build server would result in a failed build.

Currently, with the DSL tools all generated files are source controlled, and therefore other mechanisms need to be put in place to avoid the contention with locked source control files.

The following automated build approaches can be implemented to address these issues:

1.     Unmanaged – (Ignore current developer generated artifacts).

a.     Fetch latest check-in code (intermediate artifacts, and generated source artifacts)

b.     Verify intermediate artifacts

c.     Delete all previously generated source artifacts (on disk)

d.     Re-generate source artifacts from intermediate artifacts, and compile (on disk)

2.     PolicyVerification – (Verify developer transformation process was enacted correctly at check-in time)

a.     Fetch latest check-in code (intermediate artifacts, and generated source artifacts)

b.     Verify intermediate artifacts

c.     Re-generate source artifacts from intermediate artifacts in an offline temporary location.

d.     Diff re-generated source artifacts with fetched generated source artifacts. Fail the build if generated code is different (CRC check on each file, and folder content check)

e.     Compile

3.     Managed – (Synchronise generated artifacts).

a.     Fetch latest check-in code (intermediate artifacts only)

b.     Verify intermediate artifacts

c.     Check-out all generated source artifacts (cancel existing exclusive checkouts). Fail the build if not possible (due to exclusive locks)

d.     Re-generate source artifacts from intermediate artifacts, and compile

e.     Check-in all generated source artifacts

4.     UnSourceControlled – (Generated files not source controlled)

a.     Fetch latest check-in code (intermediate artifacts only)

b.     Verify intermediate artifacts

c.     Generate source artifacts from intermediate artifacts (offline temporary location), and compile

Each one of these approaches has pros and cons, and none provides and out-of-the-box solution that will work in all development environments without a policy change. For example:

·         ‘UnManaged’ always ensures the latest binaries are compiled, ignores existing generated artifacts, and does not ensure that the latest generated source code is checked-in to the source repository.

·         ‘PolicyVerification’ ensures that generated source code is correctly checked-in at development time, but cannot guarantee the developers don’t have subsequently opened new locks on generated files at the time of the automated build (although this depends on frequency and timing of the automated build).

·         ‘Managed’ ensures developers don’t have generated files checked-out, but can’t ensure they are not locked (assuming it can’t unlock them in those cases). Having generated files checked-out will be very common in cases where the build process executes during development hours. This is because executing the ‘Transform Templates’ command in Visual Studio automatically checks-out all text templates and their generated files.

·         ‘UnSourceControlled’ bypasses all issues with source controlled generated files (since generated files are not source controlled), but at the expense of source controlling all artifacts of a solution (Generated files are still considered part of the solution and the best practice is to source control all artifacts.)

One means to implement this with TFS and guarantee that all files are source controlled and that all artifacts are valid is to employ the following policies:

·         Don’t allow exclusive locks on generated files.

·         Implement the ‘PolicyVerification’ approach as a checkin-policy.

·         Implement the ‘Managed’ approach as a MSBUILD task.

In this way, the generated code will always represent the DSL definition at the time of build, and this will be synchronized to all developers. The build process should never fail even if developers have generated files checked out, since the build process can cancel the check outs. Locks are not an issue here now.

The following sections detail how to implement some of the discussed approaches for the DLS Tools using different mechanisms of build automation.

‘Unmanaged’ Approach, as a Pre-Build Event

The DSL tools provide a script ‘DslTextTransform.cmd’ (located by default: ‘%ProgramFiles%\Common Files\Microsoft Shared\Text Templating\1.0’), which encapsulates the ‘TextTranform.exe’ tool in a script for executing on specific text templates for common automation scenarios. This script, in its current form, requires a single argument to run on a specific text template, and it reads values of certain working directories and settings from the registry.

Ideally, we want to execute this script on multiple text templates. For example, all text templates in a particular project folder, and we want to provide our own values to the tool to control this transformation process.

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

With some modifications, this script can be adapted to work on the build server, for all text templates within a specific folder, and settings that are normally read from the registry, can be provided to script.

@echo off
REM %1 The folder that contains the template files that need to be transformed
REM %2 The root namespace of the project 
REM %3 The solution that gets currently build by team build
:: Call TextTransform.exe with common options required by a standard  project
 
set TEMPLATEDIR=%1\
set ROOTNAMESPACE=%2
set SOLUTIONROOT=%3
 
set INCLUDEPATH=%SOLUTIONROOT%\MySoftwareFactory\BuildExtension\TextTemplates
set ASSEMBLYDIR=%SOLUTIONROOT%\MySoftwareFactory\BuildExtension
set REFPATH=%SOLUTIONROOT%\MySoftwareFactory\BuildExtension
set TT=%SOLUTIONROOT%\MySoftwareFactory\BuildExtension
 
 
if not exist "%TT%\TextTransform.exe" (
      echo DSL Tools are not installed or installed incorrectly on this machine.
      goto :EOF
)
 
if not exist "%1" goto :usage
 
REM We call the remove function to first delete all generated files. 
call :remove
 
FOR %%A IN (%TEMPLATEDIR%\*.tt) DO (
setlocal DISABLEDELAYEDEXPANSION
"%TT%\TextTransform.exe" -I "%INCLUDEPATH%" -r 
Microsoft.VisualStudio.TextTemplating.VSHost.dll -P "%ASSEMBLYDIR%" -P 
"%REFPATH%" -dp
"DslDirectiveProcessor!Microsoft.VisualStudio.Modeling.DslDefinition.DslDir
ectiveProcessor!%AssemblyDir%\Microsoft.VisualStudio.Modeling.Sdk.DslDefinition.dll"
-a !!projectdefaultnamespace!%ROOTNAMESPACE% %%A
endlocal
)
 
goto :EOF
 
:remove 
REM To lower the risk of deleting all files from an incorrect folder 
REM we first check for the presence of an .tt file.file
REM If this doesn't exist in the folder we don't execute a delete and transformation
 
SETLOCAL enabledelayedexpansion
 
set Status=NoTTFileInFolder
for %%A in (%TEMPLATEDIR%\*.tt) do set Status=%%A
if "%Status%"=="NoTTFileInFolder" Goto :Error
 
ENDLOCAL
 
for /r %%A in (*.*) do call :goDel %%A
 
goto :EOF
 
:goDel
set ext=%~x1
 
SETLOCAL enabledelayedexpansion
 
set OkToDelete=Yes
if "%ext%"==".tt" set OkToDelete=No
if "%ext%"==".doc" set OkToDelete=No
REM add any extensions that don't need to be deleted
 
REM Execute the actual delete
if "%OkToDelete%"=="Yes" del /Q /F %1 
ENDLOCAL
exit /b
 
 
:Error
SET ERRORLEVEL=1
Goto :eof
 
:usage
echo DslTextTransform - Call TextTransform.exe for all template files in a folder
echo Usage:
echo   DslTextTransform Folder_That_Contains_tt_Templates_To_Transform
:EOF

The following changes were made to the original DslTextTransform.cmd script:

• Introduced the INCLUDEPATH variable - This variable holds the location of the default text templates that come with the DSL Tools installation. These files are referenced from the text templates in our DSL project and the TextTransform.exe needs to know where to find these.
• Introduced the ASSEMBLYDIR variable – This variable holds the location of the DSL Tools assemblies together with the assembly that contains the directive processor (Microsoft.VisualStudio.Modeling.Sdk.DslDefinition.dll ) that are also needed in the text transformation process.
• Introduced the TEMPLATEDIR variable – This variable get populated with the (first) command line parameter for this script which represents the location of the file system folder that contains the templates that we would like to transform.
• Introduced the ROOTNAMESPACE variable – This variable get populated with the (second) command line parameter which represents the root namespace of the project that that contains the template files that are transformed.
• Introduced the SOLUTIONROOT variable – This variable get populated with the (third) command line parameter for this script which represents the solution that get currently build by team build.
• Included a step to delete existing generated files - non text template (*.tt) files (i.e. *.cs, *,resx and *.xsd).
• Introduced a FOR loop that calls TextTransform.exe for every *.tt template file that can be found in the location that we pass as a command line parameter.

One means to execute this script as part of the build process is to call it in the pre-build event of all DSL projects (Dsl, DslPackage and DSL setup projects).

To configure the team build environment to use this script, we execute the following steps:

·         Copy this script to the ‘BuildExtension’ folder under ‘MySoftwareFactory’ folder in the source tree.

·         Add the following pre-build event for every folder in the DSL project (Dsl, DslPackage and DSL Setup project) that contains text templates (i.e. *.tt) files.

<PreBuildEvent Condition="'$(IsDesktopBuild)'=='false'"> $(SolutionRoot)\ MySoftwareFactory\BuildExtension\TeamBuildDslTextTransform.cmd
"$(ProjectDir)\GeneratedCode" $(RootNamespace) $(SolutionRoot)</PreBuildEvent>

‘Managed’ Approach, as MSBUILD Tasks

An alternative means to implement these approaches in the build process is to use custom MSBUILD tasks. This allows us to read the actual project structure (only deal with files added to project) and leverage other MSBUILD tasks already defined.

In the project file (*.csproj) of the DSL projects (Dsl and DslPackage projects) define the ‘<TT_Assemblies>’ property. As we are using an <ItemGroup> we can define multiple values for this property. The <TT_Assemblies> property holds the locations of several assemblies that are referenced from the ’TextTransform.exe‘ tool that we are using for the actual transformation. We also define and populate some other properties ( <TT_Directives>, <TT_Path>, <TT_IncludePath>, <TT_StrongReference>) that hold references to either assemblies that are needed or the actual command line that needs to get executed (<TT_CommandLine>).

<!-- Propery group for TransformTemplates -->
  <ItemGroup>
    <TT_Assemblies Include="$(SolutionDir)MySoftwareFactory\BuildExtension\Dsl"></TT_Assemblies>
    <TT_Assemblies Include="$(SolutionDir)MySoftwareFactory\ExternalAssemblies"></TT_Assemblies>
  </ItemGroup>
 
  <ItemGroup>
    <TT_Directives Include="DslDirectiveProcessor">
        <FullClassName>Microsoft.VisualStudio.Modeling.DslDefinition.DslDirectiveProcessor</FullClassName>      
        <AssemblyFile>$(SolutionDir)MySoftwareFactory\BuildExtension\Microsoft.VisualStudio.Modeling.Sdk.DslDefinition.dll</AssemblyFile>
    </TT_Directives>
  </ItemGroup>
 
  <PropertyGroup>
    <TT_Path>$(SolutionDir)MySoftwareFactory\BuildExtension</TT_Path>
    <TT_IncludePath>-I "$(SolutionDir)MySoftwareFactory\BuildExtension\TextTemplates"</TT_IncludePath>
    <TT_StrongReferences>-r Microsoft.VisualStudio.TextTemplating.VSHost.dll</TT_StrongReferences>
  </PropertyGroup>

We create a new <Target> called ’TransformTemplatesForTeamBuild’, and include that in the project files (Dsl and DslPackage projects). This target combines the values of the properties that we have defined earlier (in the property and item group). After that it fetches all ‘Compile‘, ‘Content’ and ‘EmbeddedResources’ items in the project that are subject to TextTransformation (AutoGen)'=='True'). Those items are linked with the ‘None’ items (based on the file dependencies) holding the TextTemplates (*.tt) . This item list is used to actually check-out the generation targets, execute the command line (TextTransform.exe) and then finally check-in the generated files. To make sure our new target gets executed we add it to the well-known ‘BuildDependsOn‘ property.

To make this task reusable over multiple project files we define this in ‘Microsoft.DslTools.targets’ file already included in our project.

<PropertyGroup>
      <BuildDependsOn Condition="'$(IsDesktopBuild)' == 'false'">TransFormTeamBuildTemplates;$(BuildDependsOn)</BuildDependsOn>
  </PropertyGroup>
 
  <!-- Transform the templates -->
  <Target Name="TransFormTeamBuildTemplates" DependsOnTargets="TransFormProperties;TransFormItems">
    <!-- Checkout the file that is generated, call the textTransform and checkin the generated file -->
    <Exec Command="tf checkout %(InternalGenerator.Identity)"/> 
    <Exec Command="$(TT_CommandLine) %(InternalGenerator.MyId)" />
    <Exec Command="tf checkin /comment:autogen /noprompt %(InternalGenerator.Identity)"  IgnoreExitCode="true" /> 
    <Message Importance="Low" Text="Transform complete"/>
  </Target>
  
  <!-- Create the properties that we need to get the correct commandline and parameters-->
  <Target Name="TransFormProperties">
    <!-- we combine the Assembly reference paths -->
    <CreateProperty Value="@(TT_Assemblies->'%(identity)',' -P ')">
      <Output PropertyName="TT_Arg_Assemblies" TaskParameter="Value"/>
    </CreateProperty>
 
    <!-- Directiveprocessors for the TextTemplates-->
    <CreateProperty Value="@(TT_Directives->'%(identity)!%(FullClassName)!%(AssemblyFile)',' -dp ')">
      <Output PropertyName="TT_Arg_Directives" TaskParameter="Value"/>
    </CreateProperty>
 
    <!-- Full commandline -->
    <CreateProperty Value="$(TT_Path)\TextTransform.exe $(TT_IncludePath)$(TT_StrongReferences) -P $(TT_Arg_Assemblies) -dp $(TT_Arg_Directives) -a
!!projectdefaultnamespace!$(DslNamespace)">
      <Output PropertyName="TT_CommandLine" TaskParameter="Value"/>
    </CreateProperty>
  </Target>
   
   <!-- Generate the list of items that are TextTransform candidates -->
  <Target Name="TransFormItems">
    <!-- Now fetch the  items that are subject for TextTemplating  -->
    <!-- Combine the items except ‘None’-->
     <CreateItem Include="@(Compile);@(EmbeddedResource);@(Content)" >
     <Output TaskParameter="Include"
             ItemName="InternalDependUpon"/>
     </CreateItem>
     <!-- cross product against ‘None’, adding the Identity, Filename and Generator as metadata -->
        <CreateItem Include="@(InternalDependUpon);@(None)" 
           AdditionalMetaData="MyId=%(None.Identity);MyFileName=%(None.FileName)%(None.Extension);MyGenerator=%(None.Generator)">
           <Output TaskParameter="Include"
                   ItemName="InternalDependAndNone"/>
        </CreateItem>
       
       <!-- Filter out items that do not depend on each other -->
       <!-- Compile uses DependUpon where EmbeddedResource and Content uses DependentUpon -->
        <CreateItem Include="@(InternalDependAndNone)" 
           Condition="('%(InternalDependAndNone.DependUpon)'=='%(InternalDependAndNone.MyFileName)' or
          '%(InternalDependAndNone.DependentUpon)'=='%(InternalDependAndNone.MyFileName)'      ) and 
                  '%(InternalDependAndNone.AutoGen)'=='true'">
           <Output TaskParameter="Include"
                   ItemName="InternalGenerator"/>
        </CreateItem>
    <!-- -->

This build task will fail when executed and any of the developers in the team have any of the generated files locked (exclusive check-out). This could be a common occurrence during normal DSL development, as developers will have to checkout generated files whilst doing text transformations on their machines. However, as long as the files remain unlocked, the build tasks should succeed. Therefore, it is recommended that a check-in policy be used with this build task that ensures all generated files are checked-in before a build, to minimize the chances of this occurrence. Please note, that even with such a check-in policy in place, this does not guarantee that no developer could have the files checked out at the time of the build.

‘PolicyVerification’ Approach, as a Check-In Policy

The site http://www.codeplex.com/sfteambuild has an implementation of a check-in policy for this verification step, as well as other useful tasks and policies to be used in similar approaches.

Automating Setup Project Text Transformation

To automate the transformation of the text templates (*.tt) in the DSL setup project we need some additional steps.

The text templates that are used by default in a DSL setup project utilize a different text transform directive processor (InstallerDefinitionDirectiveProcessor). This means we have to add another <TT_Directive> in the <ItemGroup> we defined above to reference this directive processor.

<TT_Directives Include="InstallerDefinitionDirectiveProcessor">
    <FullClassName>Microsoft.VisualStudio.Modeling.Deployment.InstallerDefinitionDirectiveProcessor</FullClassName>
    <AssemblyFile>MySoftwareFactory\Microsoft.VisualStudio.Modeling.Sdk.Deployment.dll</AssemblyFile>
</TT_Directives>

The directive processor depends on a registry key that holds the location of the ‘Microsoft.VisualStudio.Shell.dll’ assembly.

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

However in this case, this registry lookup cannot be avoided. Instead we add a new target called ‘SetVsSDKInstallDir’ to add the registry key and value that the directive processor task looks up.

By adding the ‘SetVsSDKInstallDir’ target we introduced a dependency on the ‘Microsoft.Sdc.Common.tasks.dll’ assembly from http://www.codeplex.com/SdcTasks. This Microsoft provided library contains a collection of useful MSBUILD targets including those used by ‘SetVsSDKInstallDir’.

<Target Name="SetVsSDKInstallDir" Condition="'$(IsDesktopBuild)'!='true'">
    <Registry.CreateKey RegistryHive="LocalMachine" Key="Software\Microsoft\VisualStudio\VSIP\$(VsSDKVersion)" />
    <Registry.Set RegistryHive="LocalMachine" Key="Software\Microsoft\VisualStudio\VSIP\$(VsSDKVersion)" DataType="String"  Value="InstallDir" Data="$(SolutionDir)MySoftwareFactory\BuildExtension\" />
</Target>

Since the ‘SetVsSDKInstallDir’ task creates a registry key that is not expected to already exist on the build server, due to the fact the VS SDK is not installed, it may be prudent to delete the registry key after the task has executed.

One final problem is that this directive processor has dependencies on other assemblies which it explicitly loads from the GAC (by the fully qualified name). This means that the setup project text transformation can only work for a build on a developer’s machine, but not in a team build on the build server.

Unfortunately, these hard-coded dependencies cannot be worked-around in team build scenarios in the same manner as other similar workarounds in this article due to the design of this directive processor. The only solution available to us is to copy the dependencies into the GAC of the build server. This is far from desirable considering the basic premise of this article (i.e. keeping build dependencies in the solution and not on the build server). However in this case it is unavoidable should you need to automate the transformation of the setup projects in a team build.

From a development machine with the VSSDK installed, you must copy the following assemblies from the Visual Studio SDK folder (‘%ProgramFiles%\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Common\Assemblies’), to the GAC of the build server:

·         Microsoft.VisualStudio.Modeling.Sdk

·         Microsoft.VisualStudio.Modeling.Sdk.Diagrams

·         Microsoft.VisualStudio.Modeling.Sdk.Utilities

The last dependency assembly (‘Microsoft.VisualStudio.Modeling.Sdk.Deployment’) is not deployed the same way the others are to the VSSDK local folders, and instead is deployed only to the GAC of a development machine. In order to copy that file to the build server, it must be copied out of the GAC on the local disk of development machine first (i.e. a temp folder). This can be achieved using the following command line:

cd “%windir%\assembly”
for /R %%t in (.\Microsoft.VisualStudio.Modeling.Sdk.Deployment.dll) do copy %%t “%TEMP%”

Where this assumes the GAC is installed to the default location on disk (by default: ‘%WINDIR%\Assembly’)

Automating Model Validation

DSL packages contain model definition files (DslDefinition.dsl) that the developer uses to define the DSL. The DSL implements validation rules to ensure that the domain model is correctly defined before generating any source code to implement the language and tools for displaying it. By default, validation is performed whenever the model is saved, or invoked explicitly by the developer.

Validation rules are also implemented as part of creating a custom DSL for validating instances of the designed model.

If we want to ensure that a DSL definition has been developed correctly at development time, this model definition file needs to be automatically validated by the build process. Currently, this validation step is invoked explicitly and manually by the developer when the model definition file is saved (or explicitly during development time).

Furthermore, any validation errors that are raised can actually be ignored as part of the compilation of the DSL project itself. These errors are not persisted across reloads of the DSL solution for other developers in team to address. Explicit validation is required each time the definition file is opened. In short, there is no built-in mechanism to ensure that any definition is valid at build time.

However, it is feasible to validate the DSL definition as part of a check-in policy, or within the automated build process. In either case, the definition file can be loaded into memory and the validation rules executed. The validation results (errors and warnings) can then be either conveyed to developer in the check-in policy or cause failure of the automated build process.

At present there are no tools or built in mechanisms provided by the DSL Tools to execute this validation step as part of an automated build process. Any build process would have to implement its own custom tools (MSBUILD tasks are recommended) to ensure any DSL files are valid.

The site http://www.codeplex.com/sfteambuild contains examples of such build tasks and/or check-in policies that address this requirement.

Summary

Software Factory solutions vary in structure today. Typically, they are composed of multiple project types like the ones identified in this series (e.g. VSX, GAT and DSL packages). In team build scenarios, these project types present unique challenges when automated build processes are applied to them.

Build processes and build environments on the Microsoft platform, specifically Team Foundation Server, are trending to being centrally managed and hosted in development organizations, to consolidate IT infrastructures and development project overheads. It becomes critical that these build processes and infrastructures can adapt to the introduction of newer project types, specifically of the type requiring unique toolkits and supporting resources to be present in the build process. For many organizations with these managed development infrastructures, installing dedicated SDK’s, toolkits and supporting resources (all versions of) to a centralized build server infrastructure is simply not an option on a per project basis.

In addition, project types such as GAT packages and DSL packages introduce intermediate design artifacts that require additional special verification and pre-processing steps to ensure design artifacts are valid, and all solution artifacts they generate are synchronized across whole development teams. Enforcing these verification and pre-compilation steps has significant value in a team build process.

Implementing build processes to account for these issues is neither trivial nor easily determinant without practical experience. There are many caveats, and many assumptions taken for granted in isolated (single developer) environments, are simply not valid in a managed build server environment.

This document provides the guidance to take an existing software factory solution, targeted for a single developer environment, and modify that to allow the verification and build of that solution in team build scenarios.

Whilst these modifications are rather tedious and manual today, (requiring in depth knowledge of the various toolkits and support resources), the expectation is that evolving project types such as GAT packages and DSL packages become easier to automate in team build environments in future versions of Visual Studio and Team Foundation Server. In the future, it should be possible to build a software factory solution on either a developer machine or build server without significant changes to its file structure or its project files.

About the Authors

Jezz Santos is a Principal Product Development Consultant with Microsoft Consulting Services, who specializes in Enterprise Product Development, Guidance Automation and Software Factories. His IT career has led him around the globe in several countries of Europe, America, Asia and the South Pacific. He has been immersed in Microsoft product development since the mid 1990's in a multitude of development roles for various types of companies including research, ISV, GSI, and now as a field consultant. As the result of this work, he explores the means and methods of developing and packaging reusable assets and automated guidance, making it actionable and customizable in the form of Software Factories. Jezz currently works very closely as an advisor to the Microsoft patterns & practices team, the Visual Studio Architects (VSTESA) and the DSL Tools team to help realize the concrete vision of Software Factories for Microsoft. Jezz created one of the world’s first implementations of a software factory (the ‘EFx Factory’ which demonstrates many of the advanced features of a future generation of software factories to come from Microsoft. He now spends his time helping customers and partners realize the practical vision of software factories. His blog “Software and Factories, and making sense of it for you” is dedicated to socializing and advancing fellow software developers’ understanding of this space. He is very active in the software factory building community providing guidance and various tooling innovations pushing software factory development forward. Jezz is the owner and contributor to several open source CodePlex community projects for providing tooling extensions for Microsoft software factories. You can also find him speaking at various technical events, on building software factories.

Edward Bakker is an Architect working for LogicaCMG, a major international force in IT and business services. Edward has 12 years experience in developing software and now specializes in Service Orientation, Domain Specific Languages and Software Factories. He is involved in defining the Software Factories strategy for both LogicaCMG and its customers. He works with teams within Microsoft as an expert advisor to help realize the Software Factories vision and shares his experiences with the community to support the adoption of Software Factories on real life projects. Edward is currently a Visual Developer - Solutions Architect MVP and can be contacted via his blog “Guidance, Automation and Factories”

 René Schrieken is an Architect working for LogicaCMG, a major international force in IT and business services. René has over 10 years experience in developing software and now specializes in Software Process Improvement. He defined the Software Factories strategy for LogicaCMG. He is an expert advisor for the Microsoft Service factory team. He assists development teams in adopting new technology. René in-frequently writes on his “... I'm telling you anyway!”.

-----------------------------------------------------------------------------------------------------------------------------------------------------

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication.  Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only.  MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user.  Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document.  Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

© 2007 Microsoft Corporation.  All rights reserved.

Microsoft, Visual Studio and other Microsoft products mentioned here are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

-----------------------------------------------------------------------------------------------------------------------------------------------------