A GSI's Perspective of Software Factories
by Steve Eadie
Summary: With the growing expense and complications of building software, we are continually seeking to increase our productivity while maintaining user satisfaction, but that goal is a global concern that is valid to just about any company of any size. How does a Global Systems Integrator (GSI) manage to become more agile and cost effective, while its developers may be spread across many geographic regions? How does a company know what it knows? And more importantly, how does it leverage this knowledge in a consistent way? These may seem like very strange questions and ones that would make a smaller company do a quick pulse check and wonder, "What's the problem?" The truth is that the larger the company, the more assets are collected and the easier it becomes to overlook these assets in a rush to solve a client's problems. We fail to build systems in a consistent manner. Although we can take some amount of pride when a system is delivered on budget and on time, how many projects fail even though we know what we know? Let us look at ways in which a GSI can take advantage of a software factory approach.
In today's software development landscape, the story of overruns, defects, security holes, and project failures is all too common. Software development is expensive and labor intensive, and we still end up building one-off applications, hand-stitched with minimal reuse. We already know how to exploit economies of scale to automate production; however, this exploitation does nothing for software development. It is also possible to exploit economies of scope. By reusing designs and components we can build many similar but distinct systems.
Today there are a few universal needs in our changing landscape. We have a need to deploy applications/solutions at an ever-increasing pace with solutions that are growing evermore complex as our clients demand richer features. We need to connect applications/systems in increasingly unprecedented ways, master new technologies, and as always gain a competitive edge. We need to figure out how to take a requirement and have it built into a quality system anywhere on the planet with total confidence that our customers' expectations of quality, cost, and reliability will be met. The business drivers that are pushing us toward a paradigm shift are:
- Greater agility.
- Faster time to market.
- Greater productivity.
- Better and sustainable quality.
- Greater efficiency.
- Reduced complexity.
Software factories attempt to address this paradigm shift.
First, let us define software factories. Jack Greenfield of Microsoft defines a software factory as: "a highly customized development environment and technology platform supplying key artifacts and processes to accelerate common life-cycle tasks for a specific type of solution or product" (see Resources).
Four "pillars" make up a software factory: software product lines, architecture frameworks, model-driven development, and guidance in context. These pillars combine to become more than the sum of their parts to propel us toward software industrialization (see Figure 1).
Figure 1. The four pillars of a software factory
Software factories are not a new idea but are more the pulling together of several initiatives under one roof and providing the tooling for this methodology. Work has been done in the area of software product lines by Paul Clements and Linda Northrop at the Software Engineering Institute (SEI); model-driven development, specifically domain-specific languages (DSL), by Microsoft; generative programming by Krzysztof Czarnecki and Ulrich Eisenecker; architecture frameworks from the Microsoft patterns & practices Group; and guidance in context, using Microsoft's Guidance Automation Toolkit (GAT).
A software product line as defined by Paul Clements and Linda Northrop in their book, Software Product Lines (see Resources), is: "a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way" (see Resources).
They go on to tell us that software product lines are designed to take economic advantage of commonality in systems. Product lines also amortize the investment in core assets, such as:
- Requirements and requirements analysis.
- Domain modeling.
- Software architecture and design.
- Performance engineering.
- Test plans, cases, and data.
- Knowledge and skills.
- Processes, methods, and tools.
- Budgets, schedules, and work plans.
Figure 2. Assembling a client solution from viewpoints (Click on the picture for a larger image)
There are three essential activities in fielding a product line: core asset development and product development using those core assets, and both of these are supported by the third activity—technical and organizational management. The core assets can be existing mined assets or assets built from scratch for the product line, the core assets with associated processes for each asset along with production requirements, a product line scope, and a production plan that are then used to build the product.
Architecture frameworks define viewpoints that identify and separate the key stakeholder concerns. They organize tools, process, and content by that viewpoint. They relate and integrate life-cycle phases, system components, and levels of abstraction. The architecture framework is also often described as the schema or process framework for the software factory. This framework can be used to guide the workflow and activities for the project until all of the activities are complete.
Model-driven development (MDD) is a technique to use metadata captured in models to automate software development tasks. Modeling software has long been regarded as one of the best ways to build a system, as it used to capture the intent of the system. However, models are often discarded and have been seen as second-class citizens compared to code. In software factories models are used to formally express the artifacts of a certain domain and to capture the intent that tools can interpret to provide more advanced forms of automation.
Figure 3. The factory life cycle
They can be used to create highly-focused, custom languages for specific problems, platforms, or tasks. Metadata that is captured by models is used to drive automation rather than referenced only once in a while. In software factories MDD is handled using the domain-specific language (DSL) toolkit. DSLs allow you to create a modeling environment that describes accurately the product being built and also ensures a more accurate translation of the model by team members, which translates into the code generated from that model.
Guidance in context is used to describe what to do and to provide help in doing it. The guidance is partitioned for common use cases, and is attached to steps in the process and to parts of the architecture. This guidance is context sensitive, and it only appears when you need it. Tooling such as GAT allows installable packages containing organized sets of configurable guidance assets for common use cases. The guidance also can have preconditions and postconditions to allow the workflow to vary subject to constraints.
Now that we have established what it is, what does a software factory mean for a GSI? Pretty much we are in a time of offshore/best-shore development, where the goal for large companies is to reduce their costs to remain competitive. In reducing these costs the company will move their clients' work across the globe—hopefully, without sacrificing quality. This transfer, however, is not always as easy as it seems. Architects and industry experts are a scarce resource. It can certainly be seen that there are not enough experts to go around. To make the project successful, or to bail out of a project that is in trouble, we deploy these experts to where the work is being done. This deployment is not the best use of their time, since once the development is under way they should be moving on to the next big thing on the company's horizon. Instead we find them hand holding a potentially unfamiliar development team far away from home or the client.
Software factories are a way to capture that expert knowledge and distill it so that it is available wherever the work is taking place. Our architect's requirements are baked into the integrated development environment (IDE). We have guidance, best practices, and reference implementations, test cases, and so on for the product our development team is building. The development team not only knows exactly what steps it has to take, but how and when to take them. They also get the expertise gained from every product built before using that particular factory, which adds up to quite a lot of experts being sent to help a project.
A software factory can be viewed as a series of viewpoints, and at each one of those viewpoints we can associate assets such as common services, process, guidance, test plans, and so forth. For a GSI we have these assets distributed among many teams across the globe, but by pulling together the best of those assets to populate each viewpoint our factory starts to take shape (see Figure 2).
Take a look at SharePoint Web Parts development, for example. It is possible to capture all of a company's knowledge on this site and distill that into a factory by taking the best practices, guidance, reference implementations, and test cases and placing them within an applicable viewpoint (the test viewpoint and test cases, for example). Note that if these assets do not exist then they can be mined by accurately mapping a current project and building up a first iteration of the factory. Once built, the factory can then be tested on subsequent projects and refined and extended through real-world use. Subsequent iterations of the factory will be able to reduce pain points in using the factory as well as mine rote tasks and automate those tasks (see Figure 3).
Factories can be of two types, horizontal and vertical (see Figure 4). Horizontal factories are those that help internally. They become the helper factories, those that don't target a specific industry but are broader and in general can be consumed by many other factories—for example, the Enterprise Framework Factory (EFx) by Jezz Santos at Microsoft. A vertical factory, on the other hand, is targeted toward an industry or domain—for example, finance, government, or health care. An example of a vertical factory would be the HL7 Factory talked about by Mauro Regio and Jack Greenfield from Microsoft in The Architecture Journal 6 (see Resources).
Figure 4. A Software factory (Click on the picture for a larger image)
It is also possible to capture industry knowledge in the viewpoints, too. We could state that around 70–80 percent of any client's base needs in a given industry are the same, and that the 20–30 percent difference leads to customization of the solutions to meet the individual customer's needs. Your organization probably already uses industry-level frameworks that capture the commonality among various businesses within a specific industry and handle the common portions of this information. For example, if you have been working for a specific type of client such as the government for a number of years, you may already have the necessary knowledge to capture the essence of a government application.
If you can capture the commonality of industries using their business processes and the common enterprise components, variability for a given client can be captured by mining their business rules. To provide fit and alterability requires having the right parts to allow configurable systems. Examples of these parts include common processes, common services, data models, components, and objects. To fit the customer, a system must also be extensible and interoperable with existing systems. Thus, customizability in the fullest extent means being configurable, extensible, open, and supporting interoperability with legacy and other systems.
Once the commonality and variability are defined, a software factory becomes the mechanism to deploy and execute them with high levels of productivity. This mechanism enables a software consumer to rapidly configure, adapt, and assemble independently developed, self-describing, location-independent components to produce families of similar but distinct systems. Software factories would allow the workers in the field—architects, developers, or testers—to consume assets in a more prescribed way.
You are ready to start. Here is a check list of what you will need: common processes, common services, data models, components, objects, patterns, models, and test plans, scripts, and data.
Identify where your current pain points are as a business. Do you require a lot of SharePoint skills, or is VB6-to-.NET migration becoming a challenge? Once you have identified this gap you can take a thin slice of this pain point and start to build up a factory. Start collecting the best practices for this area, and document what a group is doing while building a product. For the V1 of this product you will just be collecting factory assets. Once collected, tease out the areas of variability, and put in place your recipe for building this type of product.
After you have completed this iteration of your factory, test it out using the same team that built the V1 product. Is this iteration better? What are the problems? What are the metrics? By building the factory in iterations your success rate is higher, and you do not have to take a team away from delivering real work to find out if a factory-style approach is right for you. Factories themselves are extendable, and any enhancements to the factory would be handled by industry specialists. This approach allows for versioned factories. With every product built using the factory, the factory gets refined and improved.
Now that you have built a narrow, highly focused factory, now you want to build a larger factory to encompass more of a specific industry. Building a factory is no small task, but it need not be too onerous. The Microsoft patterns & practices Group is providing software factories of their own, such as the Service Factory, the Mobile Client Factory, and the Smart Client Factory. There are also factories such as the Project Glidepath Software Factory by Michael Lehman at Microsoft—which is designed to help ISVs build, deliver, and market applications for Windows Vista and the .NET Framework 3.0—and the EFx Architectural Framework Factory, which is a Microsoft Consulting Services offering to provide a comprehensive architectural model for enterprise-strength, service-oriented applications and services. These factories can either be used "as is," or they can be consumed by your own industry-specific factory and tailored to your individual needs. You can add your own factories to this recipe until you have covered the scope of a particular industry.
As you can see, it is not necessary to boil the ocean when building your own factory. It becomes more a process of building or consuming highly-focused factories and building them up into a larger factory to meet the business needs of your clients.
Success for a software factory can be gauged by strategic software reuse through a well-managed, product-line approach that achieves business goals listed previously. However, it is worth highlighting just how a software factory would support these goals:
- We want simple, standard solutions. A software factory supports this goal by creating the factory once and stamping out many products from the template.
- Applications are designed and implemented faster; therefore, they can go to market faster. Up to 80 percent of the work can be automated by a software factory.
- Deliver the right industry and technology expertise, at the right price, at the right place, and at the right time. Domain knowledge and expertise is baked into the software factory; the experts are at hand all the way through the development life cycle.
- High-quality applications/solutions are done right the first time. The software factory is tried, tested, and proven.
- Deliver what the customer ordered. A software factory meets the customers' business needs sooner, with less risk, and by tailoring a product as opposed to developing a custom project.
Software factories allow projects to jump-start from a well-defined baseline. Teams know what to do and how to do it. They have trusted, proven assets to help them do it quickly and correctly. This foundation makes future maintenance easier, improves quality and consistency, and also makes change impact easier to predict. New hires need less hand holding, budgets and schedules are more accurate, and a factory captures knowledge for future use.
Software factories also attempt to address the fact that there are just not enough architects to go around. Everyone gains by baking the knowledge of the experts into a factory. The company gains by having the experts working on the most important task, rather than hand holding development teams. The development teams no longer have to do so many rote and menial tasks; therefore, they are freed up to work on more interesting and challenging tasks. Of course, do not forget that clients gain by getting software built to their specifications, more rapidly, and within budget.
Steve Eadie is a software architect at EDS and is a graduate of the EDS Top Gun Program at Microsoft, an intensive three-month development program that seeks to develop the next wave of architects skilled in delivering Microsoft technologies. Steve resides in the United Kingdom, where he has responsibility for software factories and domain-specific languages. His main interests include model-driven development, software methodologies, software architecture, patterns, and mobile computing.
Carnegie Mellon Software Engineering Institute
Software Product Lines
"Design and Implement a Software Factory," The Architecture Journal, Journal 6 (Microsoft Corporation, 2006)
Czarnecki, Krzysztof and Ulrich W. Eisenecker. Generative Programming: Methods, Tools, and Applications. Boston, MA: Addison-Wesley Professional, 2000.
Jezz Santos, Enterprise Framework
The EFx Architectural Guidance Software Factory
Microsoft Visual Studio Developer Center
Domain-Specific Language (DSL) Tools
Visual Studio Team System Developer Center
Guidance-Automation Extensions and Guidance-Automation Toolkit
patterns & practices: Mobile Client Software Factory
Greenfield, Jack, Keith Short, Steve Cook, and Stuart Kent. Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools. Indianapolis, IN: Wiley, 2004.
This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal website.