Making the Business Case for Software Factories
Siemens Corporate Research
Summary: In this article, the authors discuss their approach, the difficulties encountered, and the quantitative results of building a thin, vertical slice of a system by using the software-factory paradigm. The goal of this work was to show the quantitative, measurable business value of introducing the software-factory methodology in a large enterprise development project. (15 printed pages)
Scenario: Before Software-Factory Support
The Transition to a Software Factory
The Software-Factory Implementation
Why software factories? This is a question that we frequently are asked by many of our clients when we introduce our work. While it seems obvious that increased reuse, higher level of abstraction, and increased automation provide advantages to today's text-based software-development efforts, it is not enough to justify the necessary upfront investment to upper management and other stakeholders. Therefore, we started to look for quantitative data that would clearly show the business case for using this paradigm.
After thorough research of the Internet and other literature, we were unable to find quantitative data that convincingly made the case for applying the software-factory approach. While we were able to find some data on the value of using Domain-Specific Modeling (for example, from MetaCase) or product-line engineering (CMU SEI), we did not succeed in gathering quantitative data for the significance of the combined application of product-line engineering, software automation, and increased reuse of configurable components.
Furthermore, even in discussions with Jack Greenfield, one of the godfathers of the software-factory paradigm, it became clear that there is a need for quantifying the benefits when using the software-factory approach in real industry projects.
Consequently, we teamed up with a product group that seemed to be a good candidate for the implementation of a software factory to evaluate the return on investment (ROI) during a short project within the production environment. We discuss the results of this particular project work in the following sections.
The company in question is one of the world's largest companies in the domain of electrical engineering and electronics. It employs more than 400,000 people in 190 countries, and its total sales volume in 2006 was over €80 billion, with a net income of about €3 billion. Furthermore, it's one of the largest software companies in the world and employs around 30,000 software developers, even though it is not perceived as such. Its software can be found in a wide range of products in different domains, such as power plants, industrial automation, automotive electronics, and medical systems, to just name a few.
Corporate Technology (CT) is a centralized research organization within this company; its goal is to explore new technologies and techniques that promise to ensure and enhance the competitive advantage of company products in the market place. The Vice President of Software and Engineering at Corporate Technology said about software factories: "Corporate Technology sees great potential in the software-factories paradigm and, therefore, researches this area with the goal to advance software development within the company to the next level, and to gain a competitive advantage for the company products through cutting-edge software-development practices."
The potential that we see in introducing this new paradigm to the company software is the promise to make the software development more predictable and more efficient, and to increase quality. Therefore, we did a lot of work to promote this new approach by first raising the awareness within the company. Then, we showed the value in form of a published case study (Practical Software Factories in .NET ). The results reaffirmed for us the potential of this approach and sparked interest into this approach within company business units.
Nevertheless, while the paradigm looks great on paper, the critical success factor for its acceptance and, more importantly, its broad adoption within an enterprise such as this one relies on the ability to show a quantitative, measurable business value in real-world product-development scenarios.
In our case, the criteria for choosing to implement and evaluate this particular project were:
· Availability of historical data on development efforts. This data is used to calculate the ROI for the software factory.
· Scoped, limited domain. The smaller the domain, the more efficient tool support can potentially be provided, and the less investment in development of the tool support is required.
· Part of a bigger software product line. Many instances of similar products are created from the same core assets.
· Microsoft Visual Studio 2005 development environment. The Microsoft Domain-Specific Language (DSL) Toolkit and the Microsoft Guidance and Automation Toolkit (GAT) can be used.
· Mature and well-defined software-development processes. Easy identification of reuse and automation opportunities for a software factory.
After identifying the particular project for the software-factory implementation, we used the following methodology for undertaking and evaluating:
1. Analysis of the scenario
1. Transitioning to a software factory
2. Implementation of the software factory
3. Evaluation of the project
As we mentioned earlier, the results that we present in this article are specific to this particular project. While similar scenarios can be observed in many different product-development environments, the solution might vary depending on the project specifics.
Let's take a look at the initial situation that we addressed with our software factory: In this particular project, we have one team that provides a framework (domain engineering in product-line terms) and an application-development team that builds on top of the provided framework. Both teams essentially implement artifacts manually in the form of C# source code and descriptor files in XML format. Therefore, we can see in Figure 1 that the framework-development team provides artifacts that are used by the application-development team.
Figure 1 shows that all of the information is distributed in an unstructured way within the heads of multiple developers. The developers have to express this information and put it into text-based source code. In fact, each module that is provided by the framework requires about 220 lines of source code, and each use by the application developer adds another 100 lines. Furthermore, the application developers have to be intimately familiar with the provided framework in order to make efficient use of it.
Figure 1. Scenario before software factories
In the current approach, we identified a number of problems:
· Much of the code that the framework developers create is trivial wiring code. Trivial code is no fun, costs a lot, and leads to carelessness and, ultimately, bugs.
· C# source code and XML descriptor files are tightly coupled. Changes to the source code almost always require manual changes to the descriptor files. Dependencies are easily broken by developer mistakes. Listing 1 shows an example of a descriptor file.
· The framework contains a great number of plug-ins that are very similar, yet distinct.
· Creating applications consists mostly of wiring framework plug-ins together; again, this is a time-consuming manual task.
Listing 1. XML descriptor file
<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated code! Manual changes will be overwritten. -->
<!-- Adapt the text-generation template instead. -->
<Sample xmlns="urn:sample:Common.SampleManager" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:sample:Common.SampleManager\Sample.Services.SampleManager.xsd">
<PROPERTY NAME="SampleProperty3" VALUE="test6"/>
<PROPERTY NAME="SampleProperty1" VALUE="test2"/>
<PROPERTY NAME="SampleProperty12Property" VALUE="test3"/>
Based on the analysis and the identified problems, we started performing a software-factory analysis. The goal for the analysis of the existing development process was to identify automation and reuse opportunities in the areas of framework and application development. We used an approach that was similar to the one that we described in our book Practical Software Factories in .NET  to derive a software-factory schema in iterative steps. From feature matrixes, we identified commonalities and variabilities.
The existing documentation of both framework and applications facilitated the generation of architectural viewpoints. We then added the software-development process information to the architectural description to identify opportunities for software-factory support of the project. The result of this analysis is the Software Factories Schema that we used to identify the following two opportunities for implementing software-factory support to the project by offering the biggest ROI:
1. Generation of some framework C# artifacts using a DSL—Using a domain-specific modeling language, the framework developer can use a graphical model to fill in the variables of a plug-in implementation. The implementation, including XML descriptor files, is then automatically generated from these models.
2. Support of application developers through guidance and automation—Multiple wizards developed by using the Microsoft GAT collect information from application developers and automatically create stubs and wiring code.
Figure 2 shows an overview of the envisioned software-development process with software-factory support. On the left-hand side are the actors (multiple developer types who develop the framework and the application) and the information that these actors have to provide (variability points that the developer has to fill in at modeling time). You can see that the information is very well-structured, and only very few information pieces have to be provided by the developers. Therefore, it can be entered very quickly.
The two automation mechanisms are shown in the center of the diagram: the DSL for the framework developers and the GAT package for the application developer. Artifact generators, both for DSL models and GAT recipes, automatically generate code by using Text-to-Text Transformation (T4) Templates.
Figure 2. Software-factory support
As Figure 2 shows, the developer has to add only about 10 pieces of information in this scenario. Part of the information that is provided is entered via selection from prepopulated selection boxes; others are to be entered as string. For the developer, instead of entering 320 lines of code per definition and use, only a few strings have to be entered, and some decisions that are based on valid options have to be made. The net result is not only time savings, but also quality increase by providing only valid choices and the ability for validation of input before artifact generation.
The collection of customizable integrated assets that can be installed on a developer's computer to tailor and extend the development environment to streamline specific development tasks is also referred to as the Software Factory Template. The implementation of the Software Factory Template is based on the Software Factory Schema. In our case, we provide multiple T4 templates—for artifact generation, wizards, and a DSL—that are bundled into installer packages to provide the developer with additional tools and context-sensitive automation mechanisms.
We therefore started the software-factory implementation with a DSL using the Microsoft DSL Toolkit. We chose this tool for the reason that it integrates tightly with Microsoft Visual Studio 2005, which is the mandated development tool for this project. In the first step, we developed a meta-model of the DSL in iterative steps and in close collaboration with the framework-development team. By using this DSL, developers can create model-based orchestrations using the concepts that are defined by the meta-model, which are shown in Figure 3.
Figure 3. Visual Studio 2005 toolbox
Based on the model, a T4 template engine then generates implementation artifacts in the form of C# source code and related XML files. T4 templates are a mixture of plain text and embedded instructions that enable one to embed logic, to alter the output depending on information that is provided by the user—for example, within the models or via wizards. T4 templates look very similar to ASP scripting code.
The second automation tool that is provided through the Software Factory Template is the GAT recipes for application developers. Figure 4 shows how developers can access one of those recipes—namely, the "Add Sample implementation" recipe. This opens a wizard page that collects additional information, such as the defined type, namespace, and class name.
Figure 4. Application guidance
The implementation is then generated from a combination of Visual Studio Templates that provide the project structure in Visual Studio 2005, such as class and project files, and T4 templates that provide content by filling in the information that is provided by the application developer. The business logic that developers create manually is kept in separate files, using the Partial Class feature of .NET Framework 2.0.
Lastly, we bundled the generated Software Factory Template assets into installation packages, which can be installed easily into development environments.
After the deployment of the Software Factory Template, we evaluated the software-factory ROI by first pulling out historical data on effort for the manual implementation of the assets, as it was done before the software-factory implementation. Then, we looked at the effort that was needed to implement the software factory; and, last but not least, we calculated the ROI by using the software-factory approach.
It is important to mention at this point that we compared only quantitative data that we could measure, which mainly is development effort. We did not include any assumptions about higher quality or less maintenance effort in our evaluation, to ensure that the data on ROI is as objective as possible for this project. We should mention also that the results that are presented are specific to this project only, and that other projects might, or will, offer different ROI results. Nevertheless, we believe that if the project is carefully chosen, a positive ROI can be achieved in similar industry projects.
Manual Development Effort
The historical data that was collected showed that the development effort that is necessary to implement the artifacts that are covered in this scenario was two staff days, on average. This effort was split between the different artifacts that had been implemented for Type and Resource definitions (XML files), annotations (text), task implementations (C# file), and Resource and Type PlugIns (C# files). As you can see, there were at least two XML files, three C #files, and optionally one or more annotations required. The effort also included basic debugging and testing by the developer. Additional testing by the testing team was not included in this data.
Software-Factory Development Effort
We monitored very closely the development efforts that were spent on the different parts of the software-factory implementation in order to get a precise picture of the investment, and we compared it to the one-off development scenario. The following are the results for the breakdown in the two main elements of the software factory.
Figure 5 shows an overview of the effort for the DSL implementation versus the one-off development. For the developed DSL, we spent five staff days on the domain analysis. This was basically the time to develop the Software Factory Schema. In our case, we performed an initial analysis of about three staff days. We then revisited and refined the analysis during the course of the project. The second largest account on the DSL implementation side was to develop the generators, or T4 Templates. The reason that this took considerable effort is not necessarily the complexity of the templates, but instead the fact that the debugging and editing of them basically is not supported by tools and, therefore, takes a fairly long time (trial-and-error). In our scenario, we spent another five days to develop the code-generation templates.
Figure 5. DSL development effort for software-factory implementation
Implementation of the domain concepts took another three days; these are the language elements that were shown earlier, in Figure 3. In addition, we had to define the meta-model for the DSL, which took two staff days. Because we performed a thorough domain analysis, we could implement the meta-model fairly rapidly; but we also have to admit that with the refinement of the domain analysis, we had to change our meta-model, which was sometimes cumbersome because of the limitations of the tool when we made changes to the already existing meta-model.
In our description so far, another area that we did not touch upon is the definition and implementation of constraints. Upon saving the model, we implemented checks to ensure that the model is valid and show possible errors in the model to the developer in the output window of Visual Studio. We checked the model for errors and completeness. The implementation of these constraints took two staff days.
By using our DSL, we implemented some components. Each component actually took about one hour on average, which is very little time, compared to the development effort without a DSL.
Figure 6 shows the development-effort tally for the guidance package. Development of artifacts without supporting GAT recipes takes application developers about half a day.
Figure 6. Guidance-package development effort for software-factory implementation
If we look at the "Guidance Package" column, we can see that development of the GAT activities took most of the total time. Activities are actions that are usually written in a .NET language and, therefore, can range from simple to very complex.
The guidance implementation for our project turned out to be rather simple. Altogether, it took one day to implement the activities for the guidance package that we developed. The recipes are contained in an XML file that binds together all of the activities and templates and defines how they can be accessed, and when and in which order they are executed. The development of the recipe in our project took only 0.5 staff days. The Visual Studio Template that populates the project structure and calls the T4 templates took another 0.5 days to implement. Last, but not least, we implemented multiple T4 templates that generate project items by using the information that is collected from the developer, which accounted for another 0.2 staff days. The developer effort for the implementation of the components using the guidance package shrank from the 0.5 days in the original scenario to 0.2 staff days, using the automation that was provided by the guidance package.
The Return on Investment
From the preceding sections, you can see that the investment that we made will pay off only if we are considering building multiple products by using the software-factory implementation. In essence, the software factory supports producing many similar yet distinct products, or, in other words, a software product line. Using midterm project-planning estimates, we expect that we will develop approximately 20 different components by using the DSL and at least 50 components by using the guidance package.
Figure 7. ROI for multiple products in the described scenario
Figure 7 shows the ROI. The graphs clearly show the initial investment in the automation assets of the software factory. Because of these initial investments, the marginal product cost for each following product is lower, which results in a flatter cost curve (shown in red). We can see the break-even point is at about nine products for the DSL and seven for the guidance package. If we calculate the ROI for the DSL with the previously stated assumptions, we come up with the following results:
Figure 8 shows the combined results for the whole software factory in a graph. Again, the initial investment for the software factory is high, compared to one-off implementations, but the marginal cost for each additional product is much lower. As a result, after about eight products, we reach a break-even point with the software factory that we developed. Any additional product that is developed will result in a positive ROI.
Figure 8. Software-factory ROI in the described scenario
In this article, we discussed the results of the software factory that we developed with a product group within one of the largest software companies in the world. The goal of this implementation was to show quantitatively the effects of using the software-factory paradigm on a software product in development. When starting the project, we predicted a productivity increase due to the use of a DSL by a factor of two or three. In reality, the results of the implementation turned out to be even better. The DSL development increased productivity by a factor of 10 with a ROI of more than 100 percent.
For the guidance package, we could show a ROI of more than 300 percent with minimal investment. If we consider these results and use conservative estimates, we can calculate the total effects of this software factory on a broader scale, as shown in Figure 9. Not considered in these results are clear improvements in software quality, maintenance cost, and, of course, developer motivation.
Figure 9. Estimated ROI for broader software factory
Although these results are very encouraging, there are some questions that still must be investigated. The reason for the high ROI numbers that are shown for this project was the fact that we could identify a very narrow domain, which allowed us to use commercial tools to implement the automation.
The questions that we did not touch upon yet are the following:
· How does this approach scale to broader domains?
· How does the tool support work for more complex implementations?
· How much savings will there be on maintenance efforts?
· How do we handle evolution of DSL meta-models and models?
We will try to answer these questions during our ongoing research on this and more software-factory projects in the future.
 Lenz, Gunther, and Christoph Wienands. Practical Software Factories in .NET. Berkeley, CA: APress, 2006. (ISBN 159059665X)
 Greenfield, Jack, and Keith Short. Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools. Indianapolis, IN: Wiley Pub., 2004. (ISBN 0471202843)
 Clements, Paul, and Linda Northrop. Software Product Lines: Practices and Patterns. (Boston: Addison-Wesley, 2002. (ISBN 0201703327)
 IEEE-SA Standards Board. IEEE Recommended Practice for Architectural Description of Software-Intensive Systems. New York: Institute of Electrical and Electronics Engineers, 2000. (IEEE 1471-2000, ISBN 0738125199)
 Czarnecki, Krzysztof, and Ulrich W. Eisenecker. Generative Programming: Methods, Tools, and Applications. Boston: Addison-Wesley, 2000. (ISBN 0201309777)
About the authors
Gunther Lenz currently works as ISV Architect Evangelist at Microsoft Corporation. Previously, he was a senior consultant on architecture and software factories at Siemens Corporate Research in Princeton, NJ. His current research activities focus on Model-Driven Software Development (MDSD), model evolution, and software factories. His blog is available at http://blogs.msdn.com/glenz/.
Christoph Wienands is a Software Engineer at Siemens Corporate Research. His interests are software factories and product lines, as well as service-oriented architectures (SOAs).