Gunther Lenz
Microsoft Corporation
Christoph Wienands
Siemens Corporate Research
April 2008
Summary: In this article, the authors discuss their approach,
the difficulties encountered, and the quantitative results of building a thin,
vertical slice of a system by using the software-factory paradigm. The goal of
this work was to show the quantitative, measurable business value of
introducing the software-factory methodology in a large enterprise development
project. (15 printed pages)
Contents
Introduction
Context
Scenario: Before Software-Factory Support
The Transition to a Software Factory
The Software-Factory Implementation
Evaluation
Domain-Specific Language
Guidance Package
Conclusion
References
Introduction
Why software factories? This
is a question that we frequently are asked by many of our clients when we
introduce our work. While it seems obvious that increased reuse, higher level
of abstraction, and increased automation provide advantages to today's
text-based software-development efforts, it is not enough to justify the
necessary upfront investment to upper management and other stakeholders. Therefore,
we started to look for quantitative data that would clearly show the business
case for using this paradigm.
After thorough research of the
Internet and other literature, we were unable to find quantitative data that
convincingly made the case for applying the software-factory approach. While we
were able to find some data on the value of using Domain-Specific Modeling (for
example, from MetaCase) or product-line engineering (CMU SEI), we did not
succeed in gathering quantitative data for the significance of the combined
application of product-line engineering, software automation, and increased
reuse of configurable components.
Furthermore, even in
discussions with Jack Greenfield, one of the godfathers of the software-factory
paradigm, it became clear that there is a need for quantifying the benefits
when using the software-factory approach in real industry projects.
Consequently, we teamed up
with a product group that seemed to be a good candidate for the implementation
of a software factory to evaluate the return on investment (ROI) during a short
project within the production environment. We discuss the results of this
particular project work in the following sections.
Context
The company in question is one
of the world's largest companies in the domain of electrical engineering and
electronics. It employs more than 400,000 people in 190 countries, and its
total sales volume in 2006 was over €80 billion, with a net income of about €3
billion. Furthermore, it's one of the largest software companies in the world
and employs around 30,000 software developers, even though it is not perceived
as such. Its software can be found in a wide range of products in different
domains, such as power plants, industrial automation, automotive electronics,
and medical systems, to just name a few.
Corporate Technology (CT) is a
centralized research organization within this company; its goal is to explore
new technologies and techniques that promise to ensure and enhance the
competitive advantage of company products in the market place. The Vice
President of Software and Engineering at Corporate Technology said about
software factories: "Corporate Technology sees great potential in the
software-factories paradigm and, therefore, researches this area with the goal
to advance software development within the company to the next level, and to
gain a competitive advantage for the company products through cutting-edge
software-development practices."
The potential that we see in
introducing this new paradigm to the company software is the promise to make
the software development more predictable and more efficient, and to increase
quality. Therefore, we did a lot of work to promote this new approach by first
raising the awareness within the company. Then, we showed the value in form of
a published case study (Practical Software Factories in .NET [1]). The
results reaffirmed for us the potential of this approach and sparked interest
into this approach within company business units.
Nevertheless, while the
paradigm looks great on paper, the critical success factor for its acceptance
and, more importantly, its broad adoption within an enterprise such as this one
relies on the ability to show a quantitative, measurable business value in
real-world product-development scenarios.
In our case, the criteria for
choosing to implement and evaluate this particular project were:
·
Availability of historical data on development efforts.
This data is used to calculate the ROI for the software factory.
·
Scoped, limited domain. The smaller the domain,
the more efficient tool support can potentially be provided, and the less
investment in development of the tool support is required.
·
Part of a bigger software product line. Many
instances of similar products are created from the same core assets.
·
Microsoft Visual Studio 2005 development environment.
The Microsoft Domain-Specific Language (DSL) Toolkit and the Microsoft Guidance
and Automation Toolkit (GAT) can be used.
·
Mature and well-defined software-development processes.
Easy identification of reuse and automation opportunities for a software
factory.
After identifying the
particular project for the software-factory implementation, we used the
following methodology for undertaking and evaluating:
1.
Analysis of the scenario
1.
Transitioning to a software factory
2.
Implementation of the software factory
3.
Evaluation of the project
4.
Summary
As we mentioned earlier, the
results that we present in this article are specific to this particular
project. While similar scenarios can be observed in many different
product-development environments, the solution might vary depending on the
project specifics.
Scenario: Before Software-Factory Support
Let's take a look at the initial
situation that we addressed with our software factory: In this particular
project, we have one team that provides a framework (domain engineering in
product-line terms) and an application-development team that builds on top of
the provided framework. Both teams essentially implement artifacts manually in
the form of C# source code and descriptor files in XML format. Therefore, we
can see in Figure 1 that the framework-development team provides artifacts that
are used by the application-development team.
Figure 1 shows that all of the
information is distributed in an unstructured way within the heads of multiple
developers. The developers have to express this information and put it into
text-based source code. In fact, each module that is provided by the framework
requires about 220 lines of source code, and each use by the application
developer adds another 100 lines. Furthermore, the application developers have
to be intimately familiar with the provided framework in order to make
efficient use of it.
.gif)
Figure 1. Scenario
before software factories
In the current approach, we
identified a number of problems:
·
Much of the code that the framework developers create is trivial wiring
code. Trivial code is no fun, costs a lot, and leads to carelessness and,
ultimately, bugs.
·
C# source code and XML descriptor files are tightly coupled. Changes to
the source code almost always require manual changes to the descriptor files.
Dependencies are easily broken by developer mistakes. Listing 1 shows an
example of a descriptor file.
·
The framework contains a great number of plug-ins that are very similar,
yet distinct.
·
Creating applications consists mostly of wiring framework plug-ins
together; again, this is a time-consuming manual task.
Listing 1. XML descriptor
file
<?xml version="1.0"
encoding="UTF-8"?>
<!-- Generated code! Manual
changes will be overwritten. -->
<!-- Adapt the
text-generation template instead. -->
<Sample xmlns="urn:sample:Common.SampleManager"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:sample:Common.SampleManager\Sample.Services.SampleManager.xsd">
<TYPES>
<TYPE
NAME="SampleDefine">
<TYPE_
PLUGIN
TYPE="SampleTypePlugIn"
ASSEMBLY="SampleTypePlugIn.dll">
<PROPERTIES>
<PROPERTY
NAME="SampleProperty3"
VALUE="test6"/>
<PROPERTY
NAME="SampleProperty1"
VALUE="test2"/>
</PROPERTIES>
</Sample_PLUGIN>
</TYPE>
<TYPE
NAME="SampleDefine">
<TYPE_PLUGIN
TYPE="TypePlugInSample"
ASSEMBLY="ExisitingPlugInSample.dll">
<PROPERTIES>
<PROPERTY
NAME="SampleProperty12Property"
VALUE="test3"/>
</PROPERTIES>
</TYPE_PLUGIN>
</TYPE>
</TYPES>
</Sample>
The Transition to a Software Factory
Based on the analysis and the
identified problems, we started performing a software-factory analysis. The
goal for the analysis of the existing development process was to identify
automation and reuse opportunities in the areas of framework and application
development. We used an approach that was similar to the one that we described
in our book Practical Software Factories in .NET [1]
to derive a software-factory schema in iterative steps. From feature matrixes,
we identified commonalities and variabilities.
The existing documentation of
both framework and applications facilitated the generation of architectural
viewpoints. We then added the software-development process information to the
architectural description to identify opportunities for software-factory
support of the project. The result of this analysis is the Software Factories Schema
that we used to identify the following two opportunities for implementing
software-factory support to the project by offering the biggest ROI:
1.
Generation of some framework C# artifacts using a DSL—Using
a domain-specific modeling language, the framework developer can use a
graphical model to fill in the variables of a plug-in implementation. The
implementation, including XML descriptor files, is then automatically generated
from these models.
2.
Support of application developers through guidance and
automation—Multiple wizards developed by using the Microsoft GAT collect
information from application developers and automatically create stubs and
wiring code.
Figure 2 shows an overview of
the envisioned software-development process with software-factory support. On
the left-hand side are the actors (multiple developer types who develop the
framework and the application) and the information that these actors have to
provide (variability points that the developer has to fill in at modeling time).
You can see that the information is very well-structured, and only very few
information pieces have to be provided by the developers. Therefore, it can be
entered very quickly.
The two automation mechanisms
are shown in the center of the diagram: the DSL for the framework developers
and the GAT package for the application developer. Artifact generators, both
for DSL models and GAT recipes, automatically generate code by using
Text-to-Text Transformation (T4) Templates.
.gif)
Figure 2.
Software-factory support
As Figure 2 shows, the
developer has to add only about 10 pieces of information in this scenario. Part
of the information that is provided is entered via selection from prepopulated
selection boxes; others are to be entered as string. For the developer, instead
of entering 320 lines of code per definition and use, only a few strings have
to be entered, and some decisions that are based on valid options have to be
made. The net result is not only time savings, but also quality increase by
providing only valid choices and the ability for validation of input before
artifact generation.
The Software-Factory Implementation
The collection of customizable
integrated assets that can be installed on a developer's computer to tailor and
extend the development environment to streamline specific development tasks is
also referred to as the Software Factory Template. The implementation of the
Software Factory Template is based on the Software Factory Schema. In our case,
we provide multiple T4 templates—for artifact generation, wizards, and a
DSL—that are bundled into installer packages to provide the developer with
additional tools and context-sensitive automation mechanisms.
We therefore started the
software-factory implementation with a DSL using the Microsoft DSL Toolkit. We
chose this tool for the reason that it integrates tightly with Microsoft Visual
Studio 2005, which is the mandated development tool for this project. In the
first step, we developed a meta-model of the DSL in iterative steps and in
close collaboration with the framework-development team. By using this DSL,
developers can create model-based orchestrations using the concepts that are
defined by the meta-model, which are shown in Figure 3.
.gif)
Figure 3. Visual
Studio 2005 toolbox
Based on the model, a T4
template engine then generates implementation artifacts in the form of C#
source code and related XML files. T4 templates are a mixture of plain text and
embedded instructions that enable one to embed logic, to alter the output
depending on information that is provided by the user—for example, within the models
or via wizards. T4 templates look very similar to ASP scripting code.
The second automation tool
that is provided through the Software Factory Template is the GAT recipes for
application developers. Figure 4 shows how developers can access one of those
recipes—namely, the "Add Sample implementation" recipe. This opens a
wizard page that collects additional information, such as the defined type,
namespace, and class name.
.gif)
Figure 4. Application
guidance
The implementation is then
generated from a combination of Visual Studio Templates that provide the
project structure in Visual Studio 2005, such as class and project files, and T4
templates that provide content by filling in the information that is provided
by the application developer. The business logic that developers create
manually is kept in separate files, using the Partial Class feature of .NET
Framework 2.0.
Lastly, we bundled the
generated Software Factory Template assets into installation packages, which
can be installed easily into development environments.
Evaluation
After the deployment of the
Software Factory Template, we evaluated the software-factory ROI by first pulling
out historical data on effort for the manual implementation of the assets, as
it was done before the software-factory implementation. Then, we looked at the
effort that was needed to implement the software factory; and, last but not
least, we calculated the ROI by using the software-factory approach.
It is important to mention at
this point that we compared only quantitative data that we could measure, which
mainly is development effort. We did not include any assumptions about higher
quality or less maintenance effort in our evaluation, to ensure that the data
on ROI is as objective as possible for this project. We should mention also
that the results that are presented are specific to this project only, and that
other projects might, or will, offer different ROI results. Nevertheless, we
believe that if the project is carefully chosen, a positive ROI can be achieved
in similar industry projects.
Manual Development Effort
The historical data that was
collected showed that the development effort that is necessary to implement the
artifacts that are covered in this scenario was two staff days, on average.
This effort was split between the different artifacts that had been implemented
for Type and Resource definitions (XML files), annotations (text), task
implementations (C# file), and Resource and Type PlugIns (C# files). As you can
see, there were at least two XML files, three C #files, and optionally one or
more annotations required. The effort also included basic debugging and testing
by the developer. Additional testing by the testing team was not included in
this data.
Software-Factory Development Effort
We monitored very closely the
development efforts that were spent on the different parts of the
software-factory implementation in order to get a precise picture of the
investment, and we compared it to the one-off development scenario. The
following are the results for the breakdown in the two main elements of the
software factory.
Domain-Specific Language
Figure 5 shows an overview of
the effort for the DSL implementation versus the one-off development. For the
developed DSL, we spent five staff days on the domain analysis. This was
basically the time to develop the Software Factory Schema. In our case, we
performed an initial analysis of about three staff days. We then revisited and
refined the analysis during the course of the project. The second largest
account on the DSL implementation side was to develop the generators, or T4
Templates. The reason that this took considerable effort is not necessarily the
complexity of the templates, but instead the fact that the debugging and
editing of them basically is not supported by tools and, therefore, takes a
fairly long time (trial-and-error). In our scenario, we spent another five days
to develop the code-generation templates.
.gif)
Figure 5. DSL
development effort for software-factory implementation
Implementation of the domain
concepts took another three days; these are the language elements that were
shown earlier, in Figure 3. In addition, we had to define the meta-model for
the DSL, which took two staff days. Because we performed a thorough domain
analysis, we could implement the meta-model fairly rapidly; but we also have to
admit that with the refinement of the domain analysis, we had to change our
meta-model, which was sometimes cumbersome because of the limitations of the
tool when we made changes to the already existing meta-model.
In our description so far,
another area that we did not touch upon is the definition and implementation of
constraints. Upon saving the model, we implemented checks to ensure that the
model is valid and show possible errors in the model to the developer in the
output window of Visual Studio. We checked the model for errors and
completeness. The implementation of these constraints took two staff days.
By using our DSL, we
implemented some components. Each component actually took about one hour on
average, which is very little time, compared to the development effort without
a DSL.
Guidance Package
Figure 6 shows the
development-effort tally for the guidance package. Development of artifacts
without supporting GAT recipes takes application developers about half a day.
.gif)
Figure 6.
Guidance-package development effort for software-factory implementation
If we look at the
"Guidance Package" column, we can see that development of the GAT
activities took most of the total time. Activities
are actions that are usually written in a .NET language and, therefore, can
range from simple to very complex.
The guidance implementation
for our project turned out to be rather simple. Altogether, it took one day to
implement the activities for the guidance package that we developed. The
recipes are contained in an XML file that binds together all of the activities
and templates and defines how they can be accessed, and when and in which order
they are executed. The development of the recipe in our project took only 0.5
staff days. The Visual Studio Template that populates the project structure and
calls the T4 templates took another 0.5 days to implement. Last, but not least,
we implemented multiple T4 templates that generate project items by using the
information that is collected from the developer, which accounted for another
0.2 staff days. The developer effort for the implementation of the components
using the guidance package shrank from the 0.5 days in the original scenario to
0.2 staff days, using the automation that was provided by the guidance package.
The Return on Investment
From the preceding sections,
you can see that the investment that we made will pay off only if we are
considering building multiple products by using the software-factory
implementation. In essence, the software factory supports producing many
similar yet distinct products, or, in other words, a software product line.
Using midterm project-planning estimates, we expect that we will develop
approximately 20 different components by using the DSL and at least 50
components by using the guidance package.
.gif)
Figure 7. ROI for
multiple products in the described scenario
Figure
7 shows the ROI. The graphs clearly show the initial investment in the
automation assets of the software factory. Because of these initial
investments, the marginal product cost for each following product is lower,
which results in a flatter cost curve (shown in red). We can see the break-even
point is at about nine products for the DSL and seven for the guidance package.
If we calculate the ROI for the DSL with the previously stated assumptions, we
come up with the following results:
.gif)
Figure 8 shows the combined
results for the whole software factory in a graph. Again, the initial
investment for the software factory is high, compared to one-off
implementations, but the marginal cost for each additional product is much
lower. As a result, after about eight products, we reach a break-even point
with the software factory that we developed. Any additional product that is
developed will result in a positive ROI.
.gif)
Figure 8.
Software-factory ROI in the described scenario
Conclusion
In this article, we discussed
the results of the software factory that we developed with a product group
within one of the largest software companies in the world. The goal of this
implementation was to show quantitatively the effects of using the
software-factory paradigm on a software product in development. When starting
the project, we predicted a productivity increase due to the use of a DSL by a
factor of two or three. In reality, the results of the implementation turned
out to be even better. The DSL development increased productivity by a factor
of 10 with a ROI of more than 100 percent.
For the guidance package, we
could show a ROI of more than 300 percent with minimal investment. If we
consider these results and use conservative estimates, we can calculate the
total effects of this software factory on a broader scale, as shown in Figure
9. Not considered in these results are clear improvements in software quality,
maintenance cost, and, of course, developer motivation.
.gif)
Figure 9. Estimated
ROI for broader software factory
Although these results are
very encouraging, there are some questions that still must be investigated. The
reason for the high ROI numbers that are shown for this project was the fact
that we could identify a very narrow domain, which allowed us to use commercial
tools to implement the automation.
The questions that we did not
touch upon yet are the following:
·
How does this approach scale to broader domains?
·
How does the tool support work for more complex implementations?
·
How much savings will there be on maintenance efforts?
·
How do we handle evolution of DSL meta-models and models?
We will try to answer these
questions during our ongoing research on this and more software-factory
projects in the future.
References
[1] Lenz, Gunther, and
Christoph Wienands. Practical Software Factories in .NET.
Berkeley, CA:
APress, 2006. (ISBN 159059665X)
[2] Greenfield, Jack,
and Keith Short. Software Factories: Assembling Applications
with Patterns, Models, Frameworks, and Tools. Indianapolis,
IN: Wiley Pub., 2004. (ISBN 0471202843)
[3] Clements, Paul, and Linda Northrop. Software
Product Lines: Practices and Patterns. (Boston:
Addison-Wesley, 2002. (ISBN 0201703327)
[4] IEEE-SA Standards Board. IEEE Recommended
Practice for Architectural Description of Software-Intensive Systems. New
York: Institute of Electrical and Electronics
Engineers, 2000. (IEEE 1471-2000, ISBN 0738125199)
[5] Workshop:
Guidance Automation Toolkit
[6] Czarnecki, Krzysztof, and Ulrich W. Eisenecker. Generative Programming: Methods, Tools, and Applications. Boston:
Addison-Wesley, 2000. (ISBN 0201309777)
About the authors
Gunther Lenz currently works as ISV Architect Evangelist
at Microsoft Corporation. Previously, he was a senior consultant on
architecture and software factories at Siemens Corporate Research in Princeton,
NJ. His current research activities focus on
Model-Driven Software Development (MDSD), model evolution, and software
factories. His blog is available at http://blogs.msdn.com/glenz/.
Christoph Wienands is a Software Engineer at Siemens
Corporate Research. His interests are software factories and product lines, as
well as service-oriented architectures (SOAs).