Building Software Factories - Part 1, what are we building and why?
Summary: In this part of the series we will explore what are and what it takes to build the automation parts of a software factory. We describe the various characteristics and expectations of this tool and how it manifests itself on the Microsoft platform today.
Welcome to the first part of a three part series on building software factories. In this series we are going to cover the broad topic of building software factories practically: why you would want to do that, what you need to be able to do that, and how you actually do that.
In this short series we are not going to focus on the general issues of why you want software factories, the value they bring, or how they advance the software development industry. There are many better and more detailed online resources that address those concerns and strategies and the requirements for them. This series is going to focus upon the actual practicalities of planning for, and building software factories.
This series assumes that you, the reader, already have an understanding of the benefits and value of software factories and that you have already determined that this approach might be worth trying, and you want to start to evaluate this approach practically. We are going to start with the assumption that you/your team/your organization is at the point of qualifying whether the software factories approach is the path forward for you, and whether you/your team/your organization are able to traverse that path.
This series has been written in a format that we think follows how you might think about the domain of building software factories, addressing each decision point with a qualifying question, and then answering it with practical guidance.
The series is written in three parts.
· Part 1 – “What are we building and why?” - this part, describes what a factory is in concrete terms, what it means and takes to build a factory and why we want to do that.
· Part 2 – “What is it going to take?” - tackles the critical issues of getting your organization’s management bought into a software factories approach, and what your organization needs to do to support the development of factories to give it a good chance to succeed.
· Part 3 – “How are we going to do that?” - addresses directly how to build software factories on the existing Microsoft platform, which tools, processes and practices to use to develop and release your factories, and how to get started.
Although this series is very much focused at giving you an understanding of what factories are and the process of how to execute them, it takes a very pragmatic approach to describing software factories in very practical terms. Whilst the main audience is expected to be software developers and architects, this content in not deeply technical in nature. We do not cover the implementation of any enabling software factory technologies here and there are no code samples to analyze. However, this series is targeted at and focuses upon those who will design and build software factories.
If you were looking for a formal precise definition of what a software factory is, you would find it in the definitive text on the subject, the book Software Factories written by Jack Greenfield and Keith Short. In this book, they define a software factory:
“A software factory is a software product line that configures extensible tools, processes, and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling, and configuring framework-based components.”
While very accurate, this is not really concrete enough to understand how to get started building software factories. This definition requires us to have a deep understanding of software factories and the related technologies it builds upon. From this definition it is not clear exactly where to start, with what tools and processes, and what the resultant thing looks or feels like. It describes something we’ve likely not encountered building before. The Microsoft patterns & practices team came up with a definition that describes the generation of software factories being built by them today:
“A software factory is a structured collection of related software assets. When a software factory is installed in a development environment, it helps architects and developers predictably and efficiently create high-quality instances of specific types of applications.”
This definition tells us what a factory does, but not really what the factory itself is in concrete enough terms to help us get started in building one. We need to expand these definitions with a more concrete description for the context we are going to tackle next. So, to help you understand what to build, and how, from a software factory developer’s perspective, we will describe what a factory actually is; how it physically designed; what you use to build it; and how that may look and feel as an end result. The following description is to help you understand the context of this series on building factories, and only focuses on the automation parts of a software factory. Inside the box of a released software factory you will likely find other physical guidance assets like documentation (readables), samples, frameworks (reusables), reference implementations (executables), training media, and so on. Together, with the automation parts (actionables) these will make up the total package of a complete software factory you might ship to a customer.
So, what is a software factory exactly?
It’s a Tool!
First and foremost a software factory is a software development tool. Built by developers, primarily for developers, architects, and others roles in the software development life cycle, such as designers, testers, business analysts project managers. This ‘tool’ is used to manage and automate the assembly and/or configuration of a software solution that addresses a well-known, specific business/problem domain, and is primarily used to create an executable solution from that.
How does a software factory differ from other general development tools and technologies we use today, such as C#, VB, and Visual Studio? A software factory is a specific domain focused tool, with a specific set of instructions, targeted at solving a small specific business/problem domain. Whereas, a general development tool like C# or VB is used to build basically anything we want (within limits of course) it’s a relatively unspecific tool, used to solve any business/problem domain. The point here is that a general development tool can be used to build anything, and therefore it’s left to you to manage the bounds of creating that thing (it is boundless). A software factory, through the use of a software factory schema, tooling and a managed runtime actually instructs and guides you through a known process for building the thing it knows how to build. It is very much constrained to the domain it was designed to address, and not much more.
Runs within and Configures Visual Studio
Although a software factory sounds like a new type of development tool which may have its own integrated development environment (IDE), in actual fact, Microsoft software factories are intended to extend and configure the general IDE of Visual Studio .NET. They do this in a non-intrusive and non-restrictive way that presents you with the right development tools at the right time throughout the development of the thing you are building. Incidentally, we call that thing that a factory produces a product.
For example, think of the Microsoft patterns & practices Web Service Software Factory that configures Visual Studio to develop Web services. This (early generation) software factory adds some context-aware menus, wizards, and templates to the standard Visual Studio environment (Solution Explorer) to help developers build services. The developer is still free to use all other tools like C#, XML Schema designers and so forth, but the tools for the factory (recipes in this case), extend the IDE and are only valid on artifacts that the factory builds and manages.
Future software factories will provide a much richer, more integrated design experience including more specific custom tools, designers, editors and windows etc. They will orchestrate these custom tools on a runtime that manages various views of the product, arranging those views and tools into the right layout at the right time to help guide the user through creating the specific product with the right tools.
A principal quality attribute of the software factory is that it offers its users very high productivity for creating its solutions. To achieve this, the user of the factory works with various abstractions that simplify the concepts of the business/problem domain it targets. An abstraction, in this context, is a simplified representation of something that only focuses upon the varying aspects or characteristics and variable parts of that thing that are important to those working with the abstraction. A domain model (graphical or textual) is a prime example of an abstraction in this context.
In factories, you can provide many different types of abstractions using several different types of assets like: languages, frameworks, specialized editors, wizards, recipes, models, and designers etc. This idea is really nothing new today in the Visual Studio IDE where we already use many existing abstractions to increase productivity in developing solutions: the Windows Forms Designer, Windows Workflow activity diagrams, XML schema editors, SQL, configuration files, among many others. What is new though, is providing custom dedicated abstractions tailored specifically to the product or solution we want to help build.
Although providing abstractions is one of the most important requirements of a software factory, many of today’s factories do not make extensive use of visual abstractions yet. This will be rapidly changing in the next generations of factories being built today from Microsoft. There have been past examples of powerful visual abstractions that are used in factories. For example, the ‘EFx Factory’ demonstrates some of the types of abstractions we expect to see much more of in the future. The service implementation DSL and the Product Explorer views used in this factory are ideal examples of the use of visual abstractions for the future generations of software factories.
Creates a Product
Another very important aspect of a software factory is what it outputs. A software factory typically (but not always) outputs a generated solution for the business/problem domain it targets. Like a brick-and-mortar factory, a software factory’s product is what it outputs, either as a completed product or a partially completed product that would need finalization by another tool or process (manual or otherwise). Factories are intended to output multiple product instances, each one different from the previous one based on the unique configuration of it.
A software factory does not necessarily output only generated source code files, as may be commonly observed today. For example, it’s also likely that a factory will generate some configuration files that are used to configure an existing framework, automate another system, or whatever makes sense to the solution domain it implements.
Factories can also be built that only configure or automate other systems which solve a problem in a particular domain. Microsoft Windows SharePoint Services comes to mind as one such product which defines a platform for the creation of whole collaboration applications within the product shell itself. You can imagine a software factory shipped along with this product that would be used to create to configure these collaboration applications according to some solution architecture defined within the factory. This factory may then automate SharePoint, through its exposed Web services to create collaboration application products. The factory would provide relevant abstractions of collaboration applications across whole enterprises and speed the creation and configuration of them.
One thing to keep in mind is that it is likely that a product instance will need some manual work or customization before it is ready to deploy to a particular customer scenario. This is really a side-effect of designing high-level abstractions that simplify minor details in the solution domain. To make the factory generic enough to cover all possible scenarios for the product of a factory, it is likely that minor tweaks to a product instance must be made manually to accommodate these customized solutions.
There is another type of customization that is very applicable, and potentially unique to software factories. This is the customization of the software factory itself instead of just the customization of the product (output) it generates. Factories are intended to be extensible development tools, so it’s likely that a factory user will likely want to customize the factory itself and perhaps change how it works, or the outputs its creates in some way. This type of customization, as we will see later, leads to an iterative factory development approach where we start systematically refining a factory to expand its product line (the number of variants of the products it can produce). We typically customize a factory to implement a new feature of its product, solve a defect in the factory product, or customize its physical assets to suit (for example) our development or corporate standards.
Although this series focuses mainly on the automation part of a software factory (i.e. the tooling within a factory), there are also several other types of assets that go together with the automation parts to complete the factory. Examples of these assets include: documentation, samples, reference implementations, frameworks, libraries, patterns, guides, training material etc. You will likely find all these assets included in a shipped factory to help train, educate and supplement the factory, its products and its design.
Now that we have seen some of the concrete requirements and characteristics of a software factory, it’s time to evaluate if this is the type of tool we need to build for any particular solution requirements we might have. Let’s explore the reasons you would look at building a software factory and the things you need to have in place to start that process. We will talk about the process you need to actually use to build the factory in part 3 of the series. For now let's look at when it's applicable to build one.
When is a software factory the right tool to build, to provide a particular solution? This is a particularly critical qualifying question to ensure that you pursue a software factory solution only when you have a high chance of success with building one.
The bottom line is that a software factory is not always the right answer for building any particular solution. We are going to break down the rationale for this into two sections:
· The prerequisites you need to have in place to build one
· The objectives you must have, driving you to build one.
We will then summarize by stating some of the reasons when not to build one.
To start down the path of building a factory, for the most part, you need to have done quite a bit of work up front in the domain you are targeting. You may not have done that consciously or indeed even with factories in mind, but nonetheless, you need to have made large investments in creating successful solutions for your business/problem domain already.
A Specific Domain
The very first thing you need to have is a specific business/problem domain that your factory will address, and a specific solution domain that your factory will automate. That involves scoping and constraining the business/problem domain tightly enough to warrant investment in a factory. Without a specific enough domain, you are either biting off more than you can chew, and your factory will struggle to be successful (relative to others that are more specific and specialized enough), or your domain is too generic and the value your factory can provide for solving a particular problem is too diluted (it’s usually limited by available time and resources). Either way, your factory represents your consolidated value in creating solutions to this particular business/problem domain. To be successful in the factory market, that value needs to be highly concentrated and specialized relative to others competing in this domain.
The right balance to achieve between “specific enough” and “too generic” is pretty hard to advise upon, common sense generally prevails here. As mentioned before, you certainly want to be very cautious about tackling unfamiliar domains for building your factory. Factories that tackle larger domains have proven too unwieldy to be productive and unsuccessful at providing quality implementations. For indicators of what domains are reasonable to tackle and solve with a factory, looking at existing proven frameworks or software products already available as a good indication of domain scope.
A Well-Known Domain
The second critical pr-requisite is to target a domain which is not only very specific, but which you know intimately well. It's very important you choose one you have already solved successively many times previously. This means that you already know the business/problem domain and technical solution domain intimately, and know what to preserve of that knowledge and experience (and physical assets) and how to coalesce those into a concrete tool that builds specialized products for that domain. The next steps of realizing a factory from that should be reasonably straightforward.
Since the current process of building a software factory is the abstraction and automation of existing assets, managing existing variability, and assembly of a product line of product variants; you will not know how to do this the best way without the prior knowledge and experience of doing that at least some instances of your product. Without this knowledge you are bringing too many unknowns to the table. Unknown scope, assets, variability, concerns, roles, resources, technology, patterns, markets and so on.
You would be poorly advised to attempt to build a factory for an unknown domain.
The Right Mindset
For software factory development to be a successful approach, over and above those attributes of general software development, you must have a healthy product line mentality. That is, you must have a strong affinity with the process of identifying reuse in existing solutions, and harvesting that reuse in the form of asserts/patterns/techniques from those solutions and from the process of building them. For a software factory you take those assets/patterns/techniques, generalize them, and apply them to assembling products in a product family.
In some cases, this harvesting and refinement of existing solution-based assets will not have been done prior to starting a software factory project. But as you will see in the next section, having done that process to some degree prior to building a factory is the recipe for rapid and successful factory development.
To follow on from the previous points, having assets in some form before you start building a factory is also a pre-requisite, and a natural consequence of prior solution domain experience. These solution-based assets represent your collective knowledge of the solution domain for a particular business/problem domain. Without these existing assets you have an intensive, arduous task of creating optimized, highly refined assets for your domain, which is both resource-consuming and risky.
You do not need to create all of your own assets from scratch; some of the best ones are sourced from other domain experts or groups. Your job as a factory builder is to bring them together into a cohesive form and integrate them into an automation tool to address the specific solution domain.
So what do we mean by assets? For the purposes of this series we make a distinction between solution-based assets, and automation assets.
Solution-based assets are the physical resources or artifacts you have created in solving the domain problem previously. These assets are physical things that are assembled into the product of the factory. These take many forms, but the common ones are typically: class libraries, frameworks, code/project/solution templates, patterns, documentation, and reference implementations.
Automation assets are the new things you will be developing that assemble the pieces of the solution together in your factory. These are typically: domain specific languages, models, code generators, runtimes, recipes, wizards, editors, designers, and templates.
Our basic distinction here is that it is the automation assets that provide the abstractions and assemble and configure the solution based assets into a particular solution.
To have the above domain knowledge, skills and assets, it goes without saying you must also have the time and skilled domain resources to build and support the development of a factory. These resources are, above all, absolutely required to have the domain experience and skills to follow that through.
Software factory development demands your very best dedicated, skilled resources to focus upon it.
This is because, in building a software factory, you are in fact codifying the domain knowledge and experience of those resources, and then on-selling it in the form of a tool (factory) to be used by others with lesser domain expertise. You can view your factory as containing the combined codified value present in your head/team/organization. The value of a factory therefore directly depends on what tangible value is put into the factory to start with. The success and adoption of your factory is somewhat directly proportional to the quality of the resources building it for the particular domain it targets. Unlike traditional software projects today where that value is contained within a single solution, and is often lost or irretrievable from one solution to the next, a factory is retains that value to create a whole family of those solutions.
Factory development introduces a new skill-set requirement to that of normal solution development. Like normal solution development a software factory project requires both business domain experts and technical solution domain experts to work together to build a solution that meets the business requirements of the customers. Unlike normal solution development however, factory development requires and introduces a new skill-set; that of a factory platform domain expert. This expert works between the business domain expert and solution domain expert to provide the tooling and user experience of the factory. This specific skill-set specialization requires knowledge of how to model solution domains in a way that conveys the language of the business problem and the solution being built to address that domain, and finding a balance between how the business users view the domain and how the solutions are created to solve that domain. This role is primarily responsible for defining the right abstractions and automations in the factory to present an interface and to teach the factory user to configure and assemble the solution. Traditionally in normal software development projects, a significant part of this role may be owned by the solution architect, but in factories, this also requires intimate knowledge of automation tools and technologies to describe the various abstractions in a way that is logical for the product architecture and the factory user. In today’s normal solution development projects this skill-set can be found in those who currently spend their time building application frameworks and generalizing and refactoring existing solutions into templates, frameworks and class libraries.
Of course, in most software development organizations the distribution of resources, particularly very skilled ones, is very much dependent on the organizational culture, values, middle-management support and customers. This is one aspect many organizations overlook or fail to correctly recognize to their ultimate detriment. Software factory development, like application framework development, is often viewed as an unaffordable long-term investment commitment separate from the core line of business. In a market where solutions need delivery right now on tight budgets and scare resources, this means that your business will see no immediate return on this medium to long term investment. Even then, it’s still a risk in a climate where technology is constantly advancing, and customer’s requirements are always changing. This means that choosing a software factory approach is more of a strategy that a tactic to deliver solutions today. As such, it needs understanding and support at the level of the long-range strategic planners for your company.
The net result is that unless the organization ardently protects its skilled resources and lets them focus on this type of investment undisturbed for the duration - the factory development will likely fail because the collective domain skills are slowly diluted over time, as they are moved on to solve other cases. The learning’s from those individual cases are not harvested and consolidated into central value assets, and therefore the value of the factory diminishes.
Building a software factory is no trivial task, (especially today) so, to make it worthwhile you need to be clear why you are making an investment in one. Building factories requires specific business drivers. Even with better tooling tomorrow for building your factory, you still would not create factories to build one-off solutions, and certainly not if you are not interested in having others leverage your domain knowledge and experience and physical assets.
Economies of Scale and Scope
To justify the additional expense in building a software factory (compared to a one-off development project), you need to build multiple instances (variants) of a product from your factory.
A product variant (or a product instance) is the same product created as before but with different values of its various attributes. Building many products achieves economies of scale. In the simplest sense, if we had a Web service factory, and we used it to create two different Web services, these web services are likely to be different in their data contracts or their operation contracts. These are two different instances or product variants of a Web service product. If your Web service factory always created the same Web service with same data contract and same operation contract, then you simply have a software copying tool. The same can be achieved today by simply stamping out CDROMs of the product.
However, if your Web service factory could offer variance (and a means to configure it) in the data contract and operation contract, then you can create any number of different types of Web services with it. This achieves economies of scope.
As a general rule of thumb, you will need to want to build of the order, a great deal more and no less than 5 instances of a product from your factory (during its’ useful lifetime). We will investigate more deeply how we came to this number in the following sections. But any less than this number and you will simply not be able to justify the additional cost it takes to automate the production of your products by building a factory.
Ultimately, you really need to achieve economies of scope and scale with your factory to be profitable.
Reuse and Guidance in Context
You will want to build a factory to encapsulate your expert domain guidance, experiences and knowledge, your architectures and your patterns in physical assets for reuse. This represents your value to the reuse market. You want to do this in a way that allows others to easily reuse those assets in their solution development. That way, no one will want to reinvent the wheel every time they approach this domain, instead they will use your factory and assets to solve the problem. Furthermore, to increase reusability, your assets should be “programmable” in that they can be configured (automated by the factory based on the current context of the solution under development) for use in different contexts to solve a whole class of similar problems in that domain.
You will also want to provide reuse of your assets in a way that guides the user of them in their correct use, at the correct time, for the correct purposes. In this respect, they not only provide the right tool at the right time for the right purpose, but they also teach the user how to use the factory to build the types of solution it addresses. In this teaching, the user also gains more of an insight in to the solution domain, and this aspect exemplifies the value encapsulated in the reusable asset. It is this value that the user wants to benefit from without learning the hard experiences of implementing solution for this domain.
In theory, the factory that encapsulates the best guidance for reuse ought to win the customers’ attention. (The truth of that remains to be seen of course).
One of the basic objectives of building a software factory is to provide a tool to automate design and development tasks required to build a solution in a predictable manner conforming to a known architecture (blue print). Productivity and solution transparency are the objectives here. There are many varied tasks that need to be performed in building a given solution. Without automation of some kind, you are simply providing only written guidance for completing manual steps of a process.
Full automation does not have to be the goal. In fact in most cases, as mentioned above most tasks will only ever be partially automated, and will require further configuration/customization to complete either manually or by another process. Automation can take many forms, but you will want to use automation to remove the mundane, and predictable, while preserving the creative design experience for those using your factory, facilitating the expansion of your product line. The pattern here is in feeding the factory the necessary configuration it needs to perform the various tasks on behalf of the factory user.
There is a fine balance between what can be automated and what should be configured (flexibility). In the earlier example of a fictional Web service factory, it is possible to automate the entire construction of a Web service and never allow the data contracts and operations contracts to be variable. But then we do not really achieve much variance of our products, reducing our product line, and our economies of scope. To satisfy multiple usage scenarios, we have to facilitate configuration of our products to satisfy differing requirements. However, to implement a data contract, as a factory user we should only need to provide a minimal amount of information, and then have the factory infer the rest and implement them in the solution domain.
On the other side of that coin, too little automation and we achieve little productivity. In the extreme, we essentially revert back to writing source code, unguided and from scratch again, and those using your factory will be distracted by mundane tasks and preoccupied with technical solution implementation.
Do not underestimate how potent creative thinking is when trying to find people to program computers. The future users of software factories tomorrow are the solution developers of today, they want to be lazier, not less smart.
Productivity and Usability
Productivity and high quality have to be some of the best understood larger objectives of building a software factory. To reach higher levels of productivity, you need to increase the use of abstraction and automation, and supplement that with high usability of your factory. These are the key quality attributes you need to strive for. The factory you are building is a development tool, and therefore it needs to be very usable and easily understandable for those using it. Automation should take care of the mundane tasks, so what’s left should focus on the important aspects of the product development. Usability is what is left to make that experience smooth and productive and intuitive.
Increasing the level of abstraction increases productivity through some of the following mechanisms:
· Addressing a larger, less-highly domain-skilled audience. This means more factory users are available, and able, to use your highly engineered tool to get the job done
· Providing simplified representations of the domain :
o The factory users can focus on what makes the product variants unique rather that what makes them the same.
o The factory users can focus on the important aspects of the variable parts of the product (in terms of the generally-understood domain)
o The abstractions reflect the language and vocabulary of the domain, bridging the gap between customers, their requirements, and the technical solution domain.
Abstraction and usability are key quality attributes of a software factory, and key to the adoption, the effectiveness and learning potential of any specific factory.
To collectively summarize the points above, avoid building a software factory when any of these apply:
· Confronted with a new business/problem domain.
· The solution domain is unknown or untried.
· The solution domain is broad and generic.
· You do not possess the domain skills, knowledge, and experience of this solution domain.
· You possess no existing solution-based assets, or means to obtain them.
· You cannot dedicate the most skilled domain experts to the project.
In this part of the series we have explored what are and what it takes to build the automation parts of a software factory. We described the various characteristics and expectations of this tool and how it manifests itself on the Microsoft platform today.
In order to get started building a software factory today, we need to have good understanding of the domain we are targeting and tenaciously employ a reuse mentality where we harvest the various physical assets from existing solutions of that domain for reuse in our factory. We need to recognize that a software factory itself represents the combined value of not only the physical reusable assets but also the expert domain knowledge and experience imparted by those building the factory as guidance assets.
Factory building introduces new set of skill requirements to a factory building project; those of a factory platform expert. This new role focused upon defining and applying abstractions and automation to existing solution based assets to provide a cohesive set of high-productivity and usability automation assets, and bring them together as a single integrated development tool.
Of course, this kind of new approach needs to be driven by some solid business objectives and goals mainly focused on increased productivity, reuse and conveying guidance in context with suitable application of automation. We recognized that these qualities are key to the successful adoption of our factories.
In the next part of this series (Part 2), we will explore the business case for building a software factory, including organizational changes required to support such an approach, and the costs involved in building software factories.
The final part of the series (Part 3), delves into the actual process of building a factory, and what tools and practices are available to help us get started building our factories.
Jezz Santos is a Principal Product Development Consultant with Microsoft Consulting Services, who specializes in Enterprise Product Development, Guidance Automation and Software Factories. His IT career has led him around the globe in several countries of Europe, America, Asia and the South Pacific. He has been immersed in Microsoft product development since the mid 1990's in a multitude of development roles for various types of companies including research, ISV, GSI, and now as a field consultant. As the result of this work, he explores the means and methods of developing and packaging reusable assets and automated guidance, making it actionable and customizable in the form of Software Factories. Jezz currently works very closely as an advisor to the Microsoft patterns & practices team, the Visual Studio Architects (VSTESA) and the DSL Tools team to help realize the concrete vision of Software Factories for Microsoft. Jezz created one of the world’s first implementations of a software factory (the ‘EFx Factory’ which demonstrates many of the advanced features of a future generation of software factories to come from Microsoft. He now spends his time helping customers and partners realize the practical vision of software factories. His blog “Software and Factories, and making sense of it for you” is dedicated to socializing and advancing fellow software developers’ understanding of this space. He is very active in the software factory building community providing guidance and various tooling innovations pushing software factory development forward. Jezz is the owner and contributor to several open source CodePlex community projects for providing tooling extensions for Microsoft software factories. You can also find him speaking at various technical events, on building software factories.
Edward Bakker is an Architect working for LogicaCMG, a major international force in IT and business services. Edward has 12 years experience in developing software and now specializes in Service Orientation, Domain Specific Languages and Software Factories. He is involved in defining the Software Factories strategy for both LogicaCMG and its customers. He works with teams within Microsoft as an expert advisor to help realize the Software Factories vision and shares his experiences with the community to support the adoption of Software Factories on real life projects. Edward can be contacted via his blog “Guidance, Automation and Factories”