A metaphor for the evolution of information technology into the world of service-oriented architectures.
Summary: Explores the idea that information technology is evolving in a fashion similar to how American cities have evolved over the last two centuries. The opportunities and pressures of the technological revolution have driven our metropolises to adopt new frameworks, models, and patterns for commerce and communication. Recent developments in IT are analogous. What can we learn about the present and future directions of IT by studying the recent history of our urban centers? (10 printed pages)
Cities - IT Shops
Factories and Buildings - Applications
Transportation - Communication
Manufactured Goods - Structured Data
Manufactured Assemblies - Virtual Enterprises
Retail and Distribution - Business Process
Urban Infrastructure - IT Infrastructure
City Government - IT Governance
Looking to the Future
In this paper, we are going to explore how there are similarities between the evolution of cities in the 19th and 20th centuries and the development of IT shops. In both cases, we saw gradual evolution of environments that developed in isolation. This independent development resulted in many differences of culture and how things were done.
In the middle of the 19th century, the railroads connected the majority of the cities in the United States. This resulted in people and stuff moving around in a fashion not previously possible. Moving people and stuff provided the impetus for ensuring the stuff worked together and met standards of compatibility. The changes in expectations and capabilities fueled an explosion in retailing, manufacturing, and led to the urbanization of American life.
For many years, IT shops have developed in isolation with independently developed applications and little overlap in how things were done. Since the applications weren't connected, this seemed of little consequence. Recently, it has become very practical to interconnect both the applications inside an IT shop and multiple IT shops spread around the world. People can now easily browse and visit distant applications. Chunks of data are easily transmitted to remote applications. What is still difficult is to make the data work across different applications.
Metropolis is an examination of this analogy in an attempt to both explain what is happening in Information Technology today and to show us what we can expect to happen in the future. There are eight facets to this analogy that we explore:
- Cities map to IT shops—these are systems of systems, isolated from each, trying to cope with the arrival of the railroad.
- Factories or Buildings map to Applications—the broad-stroke componentization of the isolated systems.
- Transportation maps to Communication—the impact of the railroad being analogous to the impact of the Internet.
- Manufactured Goods map to Structured Data—each changing with the arrival of standards that are so disruptive to the existing custom wares.
- Manufactured Assemblies map to Virtual Enterprises—as bicycles became assemblies of best-of-breed components, so business processes will become pipelines of best-of-breed service providers.
- Retail and Distribution maps to Business Process—interoperable metadata, such as standard sizes and ingredients lists, permit transformations and compositions.
- Urban Infrastructure maps to IT Infrastructure—as the city grows, economic pressure mounts to develop common services, such as water, sewer, and road maintenance.
- City Government maps to IT Governance—both experience the pressure to balance investment in functional growth with investment in infrastructure. Our cities transformed themselves from isolated, quirky heterogeneous systems to highly inter-operable systems in less than a century. Let us see what we can learn from their experience on our own path to streamlined interoperability!
Cities gradually evolved as sites to gather for both commerce and manufacturing. Inside the cities there were independent buildings with little or no connection between them. You might consider placement with respect to the road, but that was about the extent of the relationship to other buildings in the city.
Most of us forget today that it was a hard day's ride on horseback to go 100 kilometers. Only the most hardy and adventurous would travel that far in their lifetime. Cities had only limited exposure to each other and developed their own culture, style, and way of doing things.
Similarly, IT shops gradually evolved as new applications were built and then extended and evolved. Each application stood apart and independent of its neighbors in the same IT shop. Each IT shop had its own culture, style and way of doing things. Only the most hardy and adventurous traveler would visit multiple IT shops.
Economic pressures changed our cities. Certainly the best intentions of city planners eased the transitions—and saved some historic monuments—but economic opportunity is what really drove cities to modernize, to share services, and to devise creative means to achieve efficiencies. Economic pressures are changing our IT shops, with or without master planning. As you build new applications and 'renovate' old ones, you must consider how to link them to the shared infrastructure. How will they be connected? How can they leverage common information architecture? How will they be factored to maximize reusability? These are the challenges and trade-offs we all need to consider.
In the early part of the nineteenth century, manufacturing was typically simple and independent. The goods that were produced were limited by both the appetites of the local market and the sophistication of the manufacturing process. Factories were largely vertically integrated, producing all of the parts of the final assembly, assembling them, and even selling them. If you wanted boots you might go to the tannery. This was not the most efficient approach to the creation of goods and the manufactured items were expensive and usually not of the highest quality.
Most of our applications today are like those tanneries of the early 1800s.They produce processed data independent of each other, and deliver into limited 'markets'. They are vertically integrated and don't very often accept the work of other applications as input.
The railroad profoundly altered manufacturing. By driving down the cost of transporting manufactured parts, transportation permitted local manufacturers to produce higher quality, more sophisticated goods. Now the boots from the tannery had steel eyelets to keep the boots from tearing at the laces, and woven laces that held up better than leather thongs. Componentization allowed artisans to focus on their core competencies, rather than have to understand the diverse processes necessary to produce all of a sophisticated assembly.
Moreover, transportation made new markets available to businesses, allowing them to grow and specialize. A talented potter might come to dominate a regional market by shipping her wares up and down the rail line. ISVs—independent stuff vendors—proliferated.
For both factories and applications, independence is essential. You simply can't get any work done if you need to get everything to work together perfectly. It is only by decoupling the evolution of these pieces that you can achieve change. Yet there are inescapable advantages to interconnection. You can leverage the work of others (factories or applications) to accomplish your work. The demand of others for your work provides the economic stimulus that gives you a reason for existing.
In the middle of the 19th century, the railroad arrived! Tremendous amounts of money were made moving people, coal, and wheat. It was the delivery of people and the basic (non-manufactured) goods that drove the creation of the rails. Incredible booms and busts occurred in the 1840s and 1850s as speculators built short lines connecting a couple of cities. Eventually, these were consolidated and standards were established to allow nationwide connection by rail.
The movement of people by rail stimulated tremendous changes. People traveled to strange and wondrous places! In addition to the sweeping cultural changes brought about by travel, retail began to expand dramatically as people hopped the train to go buy stuff from other cities. Retailers were now able to gather goods into stores to offer it for sale in new ways. The movement of stuff also caused change as there were new expectations that things would work together. Before railroads it simply didn't matter if one manufacturer's goods were incompatible with another manufacturer's goods.
At the end of the 20th century, the Internet arrived! Tremendous amounts of money were made in browsing, email, jpegs, mp3s, and chat. The wires were laid to provide browsing and the movement of the simplest forms of data. The .COM boom and bust are now legendary.
People browsing changed things. The browser allowed a person to be transported to directly interact with a distant application. This direct access has driven demand for sophisticated business processing as people question why automated interaction can't replace their direct interaction via browsers. The changes from the movement of data are just beginning, though. Data still does not work together well and automated business process is still very limited.
Both transportation and communication started by moving people and the basic commodities (either commodity goods or data with simple structures). These new connections drove new changes in the standardization of stuff and data. They drove changes in retail and (soon) in business process.
In the early 1800s, goods were hand-crafted. Assemblies were created with 'trim and shim'; if the bolt of a lock didn't quite fit the slide, you trimmed a bit off with a file; if the bolt rattled in the catch, you shimmed the catch to get a tighter fit.
Pioneers like Honore LeBlanc and Eli Whitney introduced the idea of standardized parts into the manufacturing process. By establishing tight controls over the specification and production of component parts, Whitney was able to produce 12,000 muskets for the US military, effectively freeing the United States of its dependence on overseas craftsmen for its military weapons. But this was still 'in house' standardization: all of the parts were produced in one place, under one roof, and with one set of controls.
By the late 1800s, the idea had expanded across manufacturers, and de facto standards had emerged for common parts. There were sizes for nuts and bolts and cylinders, with the expectation that the ones produced by one factory would be interchangeable and interoperable with similar and complementary components produced by another. Companies that produced parts with a high degree of precision thrived; those with less consistent processes failed.
Today we still have mostly non-standard data structures. Every application models information its own way, and we depend on human operators to 'trim and shim' to integrate the applications. We see the beginnings of this movement in the XML and Web services specifications. We now have language syntax and some rudimentary rules for exchanging structured data.
Moving forward, we need to add semantics to our cross-application understanding. This will occur both in horizontal and vertical sectors. Just as the marketplace demanded interchangeability of goods in the late 1800s, it will demand the interchangeability of data in the near future.
This interchangeability will be at the level of the semantics for real business-level interaction. This means standardizing the functionality of business concepts such as customer and purchase order. Data items that will be shared across applications need standardization.
Applications must retool or perish. Organizations that fail to realize the efficiencies of integration by-design will lose in the long run to those who pursue them. The efficiencies resulting from these changes will be an economic windfall to the companies that survive the transition! Just as manufacturing standards have dramatically improved our lives, the effective use of business-level standards for data will dramatically improve our lives.
Most bicycle manufacturers do not produce tires, just as dressmakers do not manufacture their own eyelets. By creating assemblies out of best-of-breed components, individual bike makers can produce higher quality, more sophisticated products. Competition among the component manufacturers drives efficiencies and quality improvements. Bikes just keep getting better.
To do this, they need detailed specifications for the component parts. You have to match the width of the tire with the wheel with the fork with the brakes with the axle bolt. You have to consider the context in which the part will be used. Is weight or ruggedness the principle concern?
Manufacturing became a value chain driven by information, reputation, and trust. Companies partnered to change the process of bringing goods to market.
Today, companies are 'creating assemblies' of their business functionality. Rather than create a distribution and shipping department, the work is outsourced. Rather than actually building the product owned by the company, its creation is outsourced to a company that specializes in low cost, high quality manufacturing. The engineering, marketing, and ownership of the product may remain in-house (or may not remain in-house). The definition of a 'company' is evolving as surely as the definition of an assembly did in the last century.
High-speed communications and structured information offer the same promise for many other business functions, giving rise to the 'virtualization' of our organizations. A business component model can be created by clearly defining the semantics and operational requirements of our business capabilities. With clear interface definitions, we can encapsulate the details of how these capabilities are implemented. Business process becomes a traversal of these component capabilities, so each component can be orchestrated as a member of any number of processes, and may be transparently relocated inside or outside of the organization.
Value chains can be created and recreated using best-of-breed business capability providers. The 'owner' of a business process might do little more than orchestrate it. Marketing, sales, manufacturing, distribution, legal services, and human resources support might all come from specialized business service providers. The best result is agility and competitive flexibility. If distribution is backed up you just engage additional logistics providers; if your accounting service is unresponsive you replace them.
To do this, you need detailed specifications for the component capabilities. You have to match the volume of goods shipped with the countries into which you sell with the delivery guarantees consistent with the service levels your customers demand. You have to consider the context in which the capability will be used: is security or transparency the principle concern?
Standards allow the composition of stuff. Better stuff is created, because component providers can leverage the cost of optimization across a broader market. Competition drives increased efficiencies, as does a sharper focus on the unique competencies of each provider in the value chain. Business process will just keep getting better.
By the late 19th century, urban centers had developed significant retail districts. Goods had gotten more sophisticated and consumer choice had improved, but shopping was still quite an expedition. Shopping day might include taking a train into town, and then going from butcher to baker to greengrocer to dressmaker to milliner to cobbler to jeweler in search of everything you needed that week.
Stores were often still the front rooms of the factory. Visiting the shoe store might receive a request to return in hour because your shoes were not quite ready. Much of what was purchased was still custom made, and therefore quite expensive.
The new distribution capabilities enabled new approaches to retail. The standardization of sizes significantly reduced the cost of many goods by permitting mass production (while at the same time encouraging people to wear ill-fitting clothes - Most of us forget that standardized clothing was a prerequisite for daily bathing. Few could afford more than one custom-made suit and it didn't make sense to bath if your clothes were going to stink, anyway!). And the ability to transport goods to central locations for sale gave rise to the department store and the supermarket.
And then along came Wal-Mart! Wal-Mart achieved new efficiencies by wielding the power of retail over the manufacturers. Wal-Mart set the standards, not the manufacturers; manufacturers complied or Wal-Mart did not carry their goods. The effectiveness with which Wal-Mart delivered pleasant, low-cost, one-stop shopping made the store a destination for many shoppers; the power had effectively shifted to the retailer.
Now let's examine the state of the art for business process. A major innovation is swivel chair integration. Today's leading form of B2B computing is known as fax-and-pray integration. An important technique for reducing integration errors is called ALT-TAB integration which allows the use of the clipboard to copy data between applications.
If we are to do better, we need interchangeability of our data and operations. Standardized and interchangeable clothing allowed for inexpensive production of the clothes. Furthermore, this approach allowed the separation of retail and distribution from the creation of the goods. For us to make major inroads in business process we need standardized and interchangeable data and operations to allow for better composition. Then we can create computing resources and pre-allocate them for later use. This technique allows the manipulation of the resources to be handled separately from the creation of the resources. This is an essential requirement to the creation of a separate mechanism specifically for business process.
We have seen an amazing transformation in retail. People cheerfully accept standard stuff and customization is rare and expensive. But business process is still largely hand-crafted. There are poor standards. It involves a lot of 'trim-and-shim'. We have very poor 'interchangeability'.
Finally, it is clear that business process will grow to be the driving force that dictates the shape and form of applications. The work is done via business process. This will become more malleable and separate from the resources being managed. As business process becomes the economic driver, it will dictate the shape, form, and standards of applications as surely as Wal-Mart drives the standards for many, many manufactured goods!
By the late nineteenth century, the cities had grown and lots of people were living in them. Pretty soon ... it smelled bad. Urban density drove urban infrastructure. Common services such as water, sewer, gas, electricity, and telephony were built or licensed by city governments to achieve efficiencies, realize opportunities, and make the city a more livable place. These efforts required metropolitan services like dams, power plants, and sewage treatment plants (or perhaps just sewage pipes run into the bay). In addition to the metropolitan services for the infrastructure, you also needed to hook the services up to every building in the city. Running the services to the buildings was the first problem; connecting to legacy buildings was frequently a nightmare.
Many buildings were retrofitted. The Cathedral Notre Dame de Paris has flush toilets and electric lights. They were not installed when the structure was first built. The pace of change in building technologies over the last fifty years has taught us a lesson: now we put conduits from the street to the building to anticipate evolving cabling requirements.
Building and connecting to infrastructure services that are shared is usually a mix of public and private funding. Huge infrastructure projects are usually funded by the city, but individual connections are usually paid for by the businesses and homeowners. Major new private developments may not be approved unless they commit to paying for infrastructure improvements. For example, a new shopping center might have to pay for road improvements in front of the complex.
IT shops have grown and lots of applications are living in them. Some IT shops ... smell bad. Business process owners need to re-authenticate application to application. Process ownership roles are hard to map to application-specific authorization schemes. Dozens of applications produce dozens of logs in dozens of formats. Exceptions can be hard to route, parse, or use to take action. Process data is dispersed across many application-specific databases, inhibiting transparency and data mining.
Common services, such as authentication, authorization, logging, naming, and directories need to be commissioned by organizational governments, and applications need to be built or retrofitted to use them. This may require metropolitan support: without funding and a mandate from senior management, the investment is likely to be beyond what an individual process owner can afford.
Once the mandate and common services are in place, operational compliance will be required and paid for by every new application. Ease of operational integration will become a key criterion in the selection of packaged applications.
Crowded environments need well-designed infrastructure services to function smoothly. The cost can appear daunting in the short term, but the long-term cost of not doing it may drive you out of business. Just as with metropolitan infrastructure, there is competition for funding between the goals for new business functionality and the infrastructure needed to make the whole mess work well.
Cities have different visions for their shape and form. To realize their visions, governments engage in city planning. Seattle envisioned itself as a world-class city, and so engaged in bold reinventions of itself, from flattening Denny Hill, to linking the lakes to the sea, to building the space needle for the 1962 World's Fair, to spanning Lake Washington with bridges that usually float. Other cities take the opposite approach, using city planning to limit growth in order to protect livability.
To implement their vision, cities typically enlist the cooperation of both business and government. Usually, businesses are the drivers of the growth and municipalities constrain and direct the growth to match their vision.
Different visions lead to different infrastructure goals. To encourage growth, cities need to overbuild infrastructure capacity. Care must be taken to balance infrastructure investments! If a city actively pursues growth but fails to anticipate the impact on transportation, for example, congestion and inefficiency will result (come visit Seattle if you have any doubts).
Zoning, available infrastructure, and business incentives drive investment in private industry. Trade-offs are made in public/private funding according to the perceived value to the community. An easy way to start a civic debate is to propose a new sports stadium be built using public funds.
So city governments must allocate resources across an array of competing priorities, taking care to protect the sacred—such as education—and balancing the short-term, long-term, and speculative. Private industry makes decisions about what buildings to erect; constrained and controlled by planning rules.
IT governance is not so mature. Who makes the tough choices in IT? Is it the CEO, the CIO, the business unit leaders, techies, or perhaps committees? Are priorities established based on cost, flexibility, or asset utilization? What is success, and how is it measured? Are we seeking cost reductions, business process transparency, or competitive advantage?
Enterprises might learn a lot by looking at how cities manage the difficult process of resource allocation. What proposals are projected to pay for themselves? What is the timeframe and risk analysis around those projections? What in your organization is sacred? What resources remain after funding those efforts? What balance of short term, long-term, and speculative investments are right within the specific corporate culture? These problems are common for metropolitan and IT environments.
It is interesting to note that most American cities allocate resources in a way that is optimized for growth. The decision to build is usually in the hands of the business and this optimizes for growth over cost management.
So where do we sit in this timeline when we compare urban development with IT shop development?
We've seen the usage of sophisticated data structures that interchange within a single application. This is analogous to single-factory interchangeability. We've seen the Internet connect IT shops and the applications within those shops similar to the railroad's growth. It is commonplace to browse a strange and distant application just as our grandparents hopped the train for a shopping excursion and visited many different stores in a distant city. Virtual enterprises (analogous to manufactured assemblies) have barely begun. We are at the cusp of succeeding in simple business process (analogous to department stores). We are at approximately the early 1880's in urban development in our parallel IT shop's growth!
Figure 1. Development Timeline
We can see from this the innovations ahead lie in the creation of standards and interchangeability. These will allow the interconnection of interoperable pieces of computing that can move into these different roles. This will allow tremendous growth in business process and, as that business process spans companies, the increasing growth of virtual enterprises.
It is this need for independent, and yet interoperable, pieces that leads us to the service oriented architecture and the changes we see beginning in application architecture. This is not to suggest that it will take 100 more years to see a Wal-Mart equivalent, but we will the economic forces driving us to the dominance of business process over application and service standards.
We have a fun time ahead of us. We are building boomtowns with no end in sight. Sure, it won't take a hundred years for us to get there, but we won't be done tomorrow, either. The same forces that drove the maturation and sophistication of cities, civic infrastructure, and business models are driving IT today.
What do we have to look out for? How do we prepare for this adventure?
Remember that heterogeneity happens. Unless you have a very simple application portfolio, shared services will not be achieved by trying to put all of your applications on one version of one platform. Even if you could, the next merger would change that! Rather, you have to design for interoperability and integration across platforms. This is the force that is driving the industry wide work in service-oriented architectures.
IT investment is a balance of funding the sacred, protecting historic monuments, and allocating spending between infrastructure and business opportunity. Striking this balance is a key facet in effective governance, and in realizing the potential of IT in your organization.
Our standards efforts are just beginning. Most integration today is done by people. It is expensive and has high rates of error. Information integration should be a focal point for IT, since it offers such tremendous opportunity for returns on investment. Benefits will be realized through cost reduction, error elimination, more effective customer interactions, and generally better business intelligence.
Business process modeling and orchestration, too, is nascent, but will grow to become a dominant force in how applications are designed and integrated. Most understanding of organizational business processes is in the heads of the process owners. Tools and techniques for capturing and refining these processes will greatly enhance the productivity of those process owners.
You have to maintain a light hand. It is counterproductive to try to dictate what happens in every structure in town, what color shirts are made, and how much is charged for soap. You have to embrace the semi-autonomous approach to governance that is characteristic of our cities, and allow the process owners to optimize and achieve efficiencies with as few constraints as possible.
Enterprise application portfolios will undergo a significant refactoring process to embrace a model that allows more autonomous control of business capabilities. Effective modeling and a light hand on the tiller will permit dynamic, distributed organizations to create and deliver more value than ever before. In ten years time, IT shops will evolve from looking like the cities of he 1880s, to looking much more like the cities of today.
Just remember to invest in the transportation systems!
About the Author
Pat Helland has 25 years of experience in the software industry and has been an architect at Microsoft since 1994. He has worked for more than 20 years in database, transaction processing, distributed systems, as well as fault tolerant and scalable systems. Pat worked at Tandem Computers designed TMF (Transaction Monitoring Facility). He was one of the founders of the team that implemented and shipped Microsoft Transaction Server (MTS), now COM+. Pat has recently focused his thinking on loosely coupled application environments. Pat can be reached at email@example.com.
Pat gratefully acknowledges Mike Burner's assistance in preparing this paper.
This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal website.