The Evolution of Architecture with Martin Fowler
April 20, 2006
Click ARCast: The Evolution of Architecture with Martin Fowler to listen to this ARCast.
Welcome, friends, welcome back to ARCast. I am your host, Ron Jacobs, bringing you yet another episode from the Architecture Insight Conference in Newport, Wales, in the U.K. Well, I was, well, I was just really happy to have the opportunity to have somebody like Martin Fowler and Ian Robinson from ThoughtWorks come on the show to talk to me about their thoughts on which the ways that we architect systems are changing. There is evolution happening, new superior methods are evolving out of the old crud, the slime that architecture has crawled up out of and, well, I just wanted to know what these things are, and who better to ask than Martin Fowler and Ian Robinson. Let's welcome them to ARCast. [Applause]
Ron Jacobs: Well, welcome back. I am Ron Jacobs, here at the Architecture Insight Conference, here at the beautiful Celtic Manor Resort, where I am joined today by Martin Fowler and Ian Robinson. Welcome, guys, to ARCast. [Martin: Thank you.] [Ian: Thank you.] And, so, today we are going to talk about the evolution of architecture and where this is going and, so, how do you see architecture changing in the last few years? Where do you think it is going? Martin!
Martin Fowler: Umm, when I think of evolution of architecture and the focus that I am, I tend to put onto it is, what one of the some mistakes, I think that, we have tended to make, make in the industries is, to view that the good design is something that you build and then develop and then change is not something that one would expect. In fact, I think part of the underlying assumption is being that, if you design things right the first time, you don't have to change it, that's when you know you've succeeded, and increasingly people, I think, have come to the conclusion that, well, this is just a very unrealistic position to be.
So, what we said we have to think about from the beginning is, how do you constantly evolve and change the design or architecture of existing systems, and how do you make that in a graceful way of doing that, really is part of your basic way of working, and that's where I see this evolution of architecture being… being quite important, and of course then there is a meta-evolution of thinking about architecture as one of the evolutions, is to realize that evolution is important, [laughs] which is I suppose the… the… the meta-question!
Ian Robinson: So, the architecture is one that's gonna accommodate evolution.
Martin: Yeah, and thinking about changes to how you put your stuff together, and coming up with techniques that allow you to be able to make changes more easily.
Ron: Ah, OK, so this is... This is part of the push or wave, agile has become a popular thing and really, because I think in the past we sort of assumed that if we can perfectly understand the thing we needed to build upfront, then we can perfectly gather requirements and perfectly build it, but in practice that wasn't practical.
Martin: Yeah! I mean… It's a… It's a very basic level. A good example of this is refactoring, which obviously is something I've been long interested in. For a long time, people really weren't interested in how you would change an existing code base! Um… Change with code base was kind of seen as a necessary evil, um… preferably to be minimized and ALWAYS one would expect it would make things worse.
And I very aggressively combated that when I subtitled my book on refactoring, I talked about improving the design of existing code, that actually change can be a beneficial thing to the overall design. And that's had quite a significant impact. And so now we are now beginning to see refactoring tools, and here I am not going to take credit, because that's the work by people who did the tools. But I think we have changed the expectations of what people say is good. Increasingly, I see now that more able development teams are absolutely expecting refactoring to be a fundamental part of how they work.
Ron: Well, that's an interesting thought, because it's sort of a necessary aspect of maintaining the code in the long run. [Ian: Yes.] I can recall working on a product in Silicon Valley in the mid-90s, where the original portions of this product were written under Windows 1.0, and they had never bothered to refactor anything, and so the code by the time it got to 95-96 was absolutely hideous, I mean, it was almost un-maintainable, because for that very reason, that they had never thought to maintain it or to refactor.
Ian: I think they are pretty good at accommodating change during software design and development, but we're less good at thinking about the overall lifetime of a system and change coming up once something has been deployed in production, and how do we accommodate change, and how do we help support operations, and those people have to maintain and support systems. And, so, a lot of our interests is in coming up with practices, designs, architectures to support change and evolution, not just during the initial design and development, but for rather the entire lifetime of the system.
Martin: This is something that is particularly important at… I mean, at this conference, like, so many of us are getting a lot of the service-oriented architecture buzz. Well! Surprise! Surprise! Services and their interfaces are gonna have to change, and it's important to make that part of your way of thinking, and that's now one of the things that Ian is particularly interested in is… How do you think about interfaces that are kind of changeable in that context?
Ron: Well, in there is a tension there, isn't there? Because for years we have been telling people that interfaces should be immutable, that they should be fixed and once people start using them you can't change them. How do you accommodate change with interfaces?
Ian: Right! I don't think that's necessarily true any more when we're talking about the community of services, service providers, and consumers. So, I know that there is lots of interest at the moment in how we evolve and version services, I know the patterns & practices group has an interest in that.
We work with clients to develop communities of service providers and consumers, where we know that the provider needs to evolve independently of those consumers, and typically we find that despite people adhering to those four tenets of SOA, nonetheless they end up strongly coupling their consumers to their providers and very often by reflecting the entire contract inside that consumer. So, they use XSD validations and a kind of naïve serialization. Which means, whenever the provider wants to change and modify what is its interface, make extensively breaking changes, then all of those consumers have to jump at the same time.
As we have been working with clients, we have to introduce these ideas around doing just enough validation, so consumers really should be consuming only those bits of a provider they really want and intend to process, rather than naïvely reflecting the entire service within each individual consumer. And if we can bring to the surface some of these real contracts that exist between service providers and consumers, how a service has actually been consumed here and now, and if we have insight into that process, then we are better positioned to plan and manage the overall evolution of that system; so what we are talking about really is just enough validation, but also introducing a form of contract that really allows the consumers to express what it is they really want of a system, what is this functionality they want to exploit, and what they expect those contracts that the service providers exposes… what they expect those contracts consist of; and once we can have insight into these contracts, then I think we are better placed to evolve the provider independently from the consumer.
Ron: OK, I am not sure I understand! How could a service... I guess, consumer of a service consumes it without taking too much, I don't know how to even express this... [Ian: OK. All right.] Too much dependency?
Ian: So, if you think first of all about the messages that we might want to exchange between a provider and a consumer: [Ron: Yeah.] I might provide… might expose a complex message structure, it might be a pretty large message that I am sending to a number of consumers.
Now, each of those consumers might only be interested in a small amount of information in that message—typically, if they did XSD validation and they are doing all-or-nothing validation of that entire document. But if the consumer had some way of telling the provider, "Well, I am interested in these particular fields, that's the nature of my contract with you as a provider, please guarantee that in the future whatever you do, as you evolve, you maintain that very particular contract and that contractual interest, or expectations that I have with you."
And then if the provider conveyed informed of each of those sets of expectations, that come from different consumers, then they really have a total set of expectations that they have to continue to fulfill, but everything else that falls outside of that, outside that set of expectations, is up for grab, they can change it willy-nilly.
Martin: And a good example of how that works, if you think of messages being transported around those XML documents, a good way to make things break with every change is to use some system that automatically generates a class of an XSD and something of that kind when anytime you can change the document, boom, you got to rebuild the class in order to be able to consume it. So you got real version issues.
Even after three fields, what it's much better to do is to write something that says, "OK, here is three bits of XPath to tell me the three bits I want to get. And I'll take that, that, and that." And then you got a much more flexible approach. If you remove fields that you are not interested in or add extra fields, the things still work. Because you are just slicing into the document and grabbing what you need, as opposed to trying to process the whole thing.
Ron: What's interesting about this is, it sort of turns the model of contracts on its head. Right! I mean, because emphasis has always been on the contract from the servers' or the service providers' point of view, saying, "Here is my contract," and you are suggesting that it ought to be possible for the clients to say, "Well! I know you have that message, but I only care about this part."
Ian: Yeah, and we call that consumer-driven contracts.
Ian: So, to my mind, services are of no use to business unless they are being consumed. So, by trying to focus on those consumers and allowing them to express what functionality they want to their service, what business value they want to exploit, then we start really getting closer to the way in which your system is realizing some business value.
Ron: But, now, the interesting thing from services' point of view there would be that the parts of my contract that matter essentially become the union of the elements that consumers care about, and anything that no consumer cared about doesn't really matter.
Ian: Exactly. And, so, I may very well publish an interface that several methods, several different messages, documents, whatever, those bits of that interface or that contract that nobody is using, well, if I got rid of that, who would notice? So, that's the stuff about interfaces being immutable, I think, there are ways of expressing our insights into the relationship between providers and consumers that the change or attitude to interfaces.
Ron: OK, I understand what you are saying, but I am not sure I see the value in thinking about it this way. Maybe, so... I guess there is value in the sense that if I wanted to add something new to my contract that no consumer knows about, therefore they don't care about it, therefore it shouldn't really matter, it shouldn't break them.
Ian: And that's… That's exactly the point, a very simple change to interface is you want to add a new field, [Ron: Yeah.] maybe because this one consumer here wants to have a field, now that shouldn't affect 50 other consumers that I have currently got. They should just carry on working, there should be no change.
Ian: Now, as I said, if we define our contracts in terms of XSD, now we got to have either up the new version number or create a whole new service just in order to accommodate with one little field that one guy wants . But if you think of your contracts in this consumer-oriented way, then nobody cares. You just add what you want for one guy.
Ron: But this kind of works against the whole flow of the way the tooling around all this stuff works, though. Doesn't it? Because all the tooling is like pointing at your contract or generate a proxy in a very fixed on that contract, but if you change that contract, everything is gonna blow up.
Ian: Yeah, but I would say that a lot of the Microsoft tools—WSE, Indigo, even ASMX Web services—allows us to inject different kinds of validators into the pipeline. So, we can do just enough validation, rather than consuming an entire service—all-or-nothing consumptions of service. So, I think we do have a lot of flexibility at the level of their technologies we are using.
Ron: Well, it depends on how advanced you want to get. Right! And, so, the very simplest way is to do these things are always going to be very tightly bound to the contract, or at least they are today.
Ian: Yeah, that's true. [Ron: Yeah.] That's fine in many, many circumstances; but, well, we know we have a community of services where we want some of them to evolve or we know that some of them are going to evolve at different rates, then I think at that point it's worthwhile considering some of these other issues and thinking about consumer-driven contracts.
Ron: And how does the consumer or how does this knowledge of the consumer cares about flow back to the service provider?
Ian: OK, so, we have expressed this in terms of a pattern, and we have tried to remain relatively implementation-agnostic. So, you could do an entire out of bound. You know, you should get team sitting next to one another. They could actually turn to one… one team member turn to another and say, you know, These are the things we are really interested in or we are really using, but you can obviously work your way up the stack and start actually passing messages out of bound. And there is provision for that in many ways, metadata exchange and stuff like, that give us some model for being able to exchange lots of additional information about the way in which services are being consumed. [Ron: Interesting.]
Martin: Another good way to think of it is in terms of example approach in one that particularly uses tests, if I require these three fields to be present on a message while I can write test cases that express that, and then send those test cases back to the provider and basically say, "As long as those test cases run, I am gonna be OK." And, so, then the provider effectively has to look at the union of test cases of the consumers. So, those kinds of techniques can also be used to try and do this kind of thing.
Ron: OK. This is one way in which we are supporting the requirements for evolution of the services, by maybe not being so tightly bound to the message contract from the consumer side. I think some other way you guys are thinking about architecture evolving to support this sort of rapid change?
Martin: Another area that… that could… In particular, a colleague of ours has been working in is of evolving databases. Because databases have traditionally been considered to be one of the things that you just got to get right from the beginning. And Pramod Sadlage is the name of my colleague who is doing this kind of stuff since about 1999, where we worked on the project in the States.
And just last week, he and Scott Ambler released a book in my series perhaps on refactoring database, which is all about how do you make these concern changes on databases and a lot of this kind of stuff is very similar. You want to be able to do things like add a column to a table, and that's obviously because of the nature of SQL itself is usually a nondestructive change, because if you add a column it doesn't affect any of your SQL calls, because you either a name what you have or you take it all that with *, and that has that same property, but you are not breaking that was built in SQL.
But, obviously, one of the complications with database is you also have to migrate the data. So, refactoring of a database also has to include data migration as a part of that. And these techniques are not been terribly well known, but we been steadily using them up onto our projects so over the last four or five years, and that to me is another very significant shift in enterprise systems is for increasing classes of application we can deal with refactoring the database.
Now, one of the big issues lies when database itself is being used as an integration mechanism. I mean, this is where, again, I think the service-oriented architecture being actually works kind of nicely. Because at least one of the common assumptions between service-oriented architecture is you should hide the database stuff behind a different kind of middle-message-oriented whatever and API, and then you don't use a shared database as an integration mechanism, use some kind of messaging techniques. So, that fits in well with that.
Ron: Well, I think one of the enlightening things Ian and you mention that you were involved in is the integration-patterns work, and when I was in patterns & practices, we used that. It was a wonderful work, highly recommended. One of the enlightening things to me about what I learned from that project as, as people have moved more towards service-oriented integration as the key way, the prime way they integrate system, often they found that sometimes the sheer volume of data that must be integrated sort of overwhelms the technology stacks that support service-oriented integration. Right now that so there are at least two different styles of integration, such as shared database.
Ian: Yeah, see, you end up with several different styles within any one system. I think that's fine. You know, we deal all the time with heterogeneous system and that's no problem, but if we need to have a certain awareness of what we are doing and insight into it, rather than having the technology lead us. So, yeah, it's a matter of evolving the practices and the insights, so that we have more confidence as we are moving into production and supporting and operating systems.
Ron: Well, you find that someone will come up and say, "Oh! I am trying to integrate these two systems and it worked pretty well in a test environment, but once we started moving, you know, millions of rows a day, the day we begin to fall away pretty quickly." And that, you know... This is when you realize that the expense of this serialization to XML and back is very, very significant when it comes to the sheer volume of data.
Ian: And that's… That's probably going to become an increasing problem, I think we are going to see more and more instrumentation injected into those services, as we are exchanging documents and messages. And, so, there is inevitably going to be an overhead, not just about serializing data, but inspecting it as we receive it or as we send it. And, so, yeah, we need to start thinking about those issues, as well.
Ron: OK. So we talked about a sort of refactoring services and refactoring databases. What else is evolving in architecture?
Martin: [Ian: Come on.] Those of the two, I guess, are very much in the front of my mind. But I think more with it, we just have to look at that as a constant statement we have to look at. Whenever we are coming up with the design about anything, we have to say "OK, what's version 2 and version 3 gonna look like, and how are we gonna get there in the graceful-est way we can do that?" And that's one of the things I like about iterative development, because it kind of forces you to evolve all the time, even in your development stage.
And one of the nice things about evolutionary of the database stuff was Pramod would do this every single iteration. It evolved the database. It migrated all the test data. He will do all of this every… months before things went live, and what happened, of course, is the… When the system went live and he had to evolve live database, he'd already figured out how to do it. With me, it was already a done thing.
So, in many ways, iterative development forces you into this constant evolution notion and, yeah, it causes a lot of pain, because you have to solve all the problems about evolve… evolution early on in the project, but early pain is cause of benefits of agile development, as far as I am concerned. [Laughter]
Ron: Well, the point is that we are going to have pain anyway; we might as well get over with it early on. I mean, you know, you talk to people about evolving a database, especially a production database; the pain is in that data migration. Just simply adding a column or whatever, database might turn for a couple of days while it migrates over the data. So, you do have to have a specific plan about how you are going to accomplish that.
Martin: [Ian: Embrace pain.] [Laughter] Yeah, what… They can get pain early, while you're still able to deal with it. The worse thing to do is to have it when you are in a rush and you got a hell of a lot of things to do, and also do little bits of pain at a time, so it gradually builds up and you learn techniques, you can apply those techniques steadily, and then you'll figure out how to cope, and it's just the same is with integration. Integration used to be a big pain for lots and lots… and it still is, of course, for lots and lots of projects.
I mean, for us at ThoughtWorks, integration has been a nonevent for many years, because we've learned to integrate all the time, multiple times a day. We had a hand, obviously, in developing Cruise Control and Cruise Control.Net, and lots of projects all over the world now use these tools and do continuous integration. And the big pain of integration is you anticipate so many facts now, as a result of that, and I think that's a… That's another great example of, "If it's painful, do it more often."
Ron: [laughs] What I can recall early in my development career, I worked on a very distributed team, and so we would often be, you know, these guys over here doing their little bit, those guys doing their little bit, we would all have our thing working, and then every couple of weeks we'd go through this very, very painful process of everybody trying to put all the changes together, and suddenly nothing would work. For two, three, five days, you know, nothing is working, and then we'd get going again. That was, that was extraordinarily unproductive.
Ian: Everything stops, and even at the end of five days, you're not entirely sure where you stand, things are working, but does it support the level of functionality you expected it to be before you tried to integrate those two systems, or have you introduced other problems that are still to pop-up further down the line?
Ron: Yeah, and, and regressions were always occurring, because one side will get the fix in, but the other side wouldn't, it in the integration. One... The good set of changes would get thrown away, and they will always come back, and lots of pain will definitely evolve around that.
Ian: Martin has already mentioned Cruise Control and continuous integration, but another colleague here, Graham Tackley, has been doing a lot of work around deploying into virtual server environments, so every day, in addition to continuous integration, we will also be deploying into environments that try to reproduce the production environments. We can tear them down and redeploy into them over and over again. So, again, we're trying to avoid being surprised by changes between development and test environments and then production environments.
Ron: Well, and I think that's absolutely one of the best thing that's happened at that long time is the ability to virtualize the production environment, especially when you are involving other parties in your development process. Because they can actually reproduce the production environment in their location, which is something that was virtually impossible before, and you would never know everything is going to work until you go in the real environment.
Yeah. So, let me ask a little bit about patterns, because this is one of the things you are well known for. And I am always grasping about how are patterns really, really helping people, architects, in the real world. I mean, other than that, we know a lot of the people have bought the books and they have seen them on their shelf. But when do they become, how do you make them actually helpful in your daily work?
Martin: Of course, I am not the right person to ask that question to. Since I am the person who writes about them and Ian maybe is the better person, since he actually has to use the stuff. [Laughter]
Ian: Well, I think, firstly, good patterns are really after the fact, they are expression of experience. So, I am always wary when somebody comes up with a new pattern that probably hasn't been tried and tested. So, good patterns are a reflection of what we've been doing for a long time, we… you know, software-development community, and so distilled experience, distilled knowledge, and we end up with a way of describing systems that don't necessarily have to go all the way down to the technical level, to the implementation detail.
But if something that helps us to bridge the gap between business and the functionality we are trying to provide to that business, I think it introduces a middle ground into the picture and a way of talking about systems that can involve a broader range of stakeholders. So, there is a language and a vocabulary that day-to-day helps development teams talk to all the other stakeholders in the program… you know, there is… a lot of a… very specific patterns that for the raising, but a lot of integration patterns in particular help us describe what is the way anticipating doing as we connect systems, and they do make sense of what we are doing to the business.
Ron: OK, so, you know, a pattern is putting a name on something that we've already been doing anyway, but, but to many people who haven't been doing that thing mean the hope, I think, is that they can read about it and learn about it and, and then suddenly they are benefiting from the collective experience of those who did it.
Martin: That certainly has been the intention, is to try and say, "Here is what we currently understand about how this pattern works and when you should use it." And that's something that people can use as a starting point, so they don't begin from a blank sheet of the paper. They can actually begin from some degree of guidances as to how to do things.
It definitely will never cover 100 percent of circumstances, because there is so much stuff in the localized things. So, one of my taglines is, "Patterns are all half-baked, you have to finish them off by yourself." But the point is, you are starting at least a halfway along, you got some idea what you got to worry about, some sense of some of the issues to deal with and how you can deal with them, and it will you help you to get there a little bit quicker and… And I am not the someone who looks for the huge silver-bullet victories, those tiny little victories please me. So, that's why I tend to be comfortable with patterns, because it's lot of little victories.
Ron: Yeah, well, and you know the thing we always struggled with when I was in patterns & practices team is thinking about, "OK! We created a book, like integration patterns, but how do we help drive that experience into the day-to-day work of the people who have to get these things done?" And probably the easiest way is to take a pattern, patterns, the patterns and bake them into the framework that the people can just apply, but that quickly becomes difficult, because the frameworks begin to grow exponentially and become very complex beasts on their own. And we were after the silver-bullet victories, like you said, and sometimes they are very difficult to achieve. Whereas, you know, just taking a small pattern, understanding and applying it, is really the goal.
Ian: Plus, patterns is simply an invitation to experience the pain earlier. [Laughter] You get excited.
Ron: Everybody is after pain-free development. But we haven't gotten there yet, aren't we? [Laughter]
Martin: Oh! We never will! Software development is tough, and I don't think anything is going to make it easier. I mean, certain things will become easier… what, memory management, for instance, but memory-managed system make that much easier, but then we got whole new problems to deal with, as well, because we always wanted to do more with software.
Martin: So, I think as a fundamental activity, software development is always going to be hard and always going to benefit most of having talented people working on it. What we can do is try and accelerate the process as much as we can, make the whole thing as rapid… And have new and different kinds of pain, instead of the old ones.
I am very happy to no longer have to manage my own memory. Thank you very much. That's great. But it doesn't mean that there is a lack of things to do. [Ron: Yeah.]
Ron: Well, Martin and Ian, thank you so much for joining me today on ARCast.
Ian: Thank you.
Martin: Thank you!
Martin Fowler and Ian Robinson, ladies and gentlemen. Hey! Wow! Thinking about evolution of architecture. You know, these guys are some really smart thinkers, and I love the work they've done with patterns. And, you know, I've encountered ThoughtWorks through my time with patterns & practices, and we collaborated with them around many projects, and they were just terrific, and you can tell they really know their stuff and are really thinking out on the edge, where things are going, and this is just fantastic stuff. So, you know, I encourage you to go to take a look at their site on http://www.thoughtworks.com. They have some interesting papers and some patterns. Ian gave me a paper of some other patterns that he has done, and that's great, and that's why we are here. Here on ARCast.