The Value of Maintenance
Summary: The purpose of this article is to convey the importance of maintenance to the application architect, and to make mention of the importance of recognizing the value of those who provide it. (6 printed pages)
"Hi, Don. How's the work on the new accrual routines going?" I had just noticed our newest architect at the water cooler, and I thought I'd check up to see how he was handling one of his first important duties.
"Oh, fine. I just headed a review and sent out the code drops this week, but some of the guys in the field aren't quite up-to-speed with it. Some of those guys are still living in the dinosaur ages."
I paused for a minute to think about what I had just been told. Don and I worked for Datum Corporation, where we wrote core banking software that was hosted in seven mid- to large-size institutions. Each bank had its own copy of our core software, but each applied customizations as it wanted.
"What do you mean? Did you get feedback on the latest code shipments?" I was asking about the reaction to the latest code updates that we had shipped to the data centers. We had been issuing major releases every six months or so, and minor revisions about every month.
"Yeah. Most of them were just fine with it, but a couple of those guys wrote me some e-mails about not understanding the code. I guess they've never seen the Mingleton pattern before; but I suppose, if you're just a maintenance drone, you don't bother to keep up on the latest development techniques. Not every coder's a genius, I guess."
I swallowed a drink of water to keep from betraying my feelings. I knew an important lesson could be learned here; but a lecture would seem condescending, and I probably would fail to get my message across. I thought for a minute about haranguing Don on the dangers of using an obscure pattern like the Mingleton. Patterns can be useful (everyone loves templates that can solve design problems for you), but there is a danger in using obscure patterns. As soon as you leave the handful of patterns that have been institutionalized in a few famous books, you run the risk of adding complexity and confusing your design with a pattern that nobody would recognize; and recognition is the value that a pattern brings! I thought it better to let that lesson remain for another day, so I punted for the meantime.
"Well, I'm not up on all the latest design patterns, myself. Maybe, I'll have a look over that code to get up to the state of the art."
I thought it over for a while, thinking that this was an excellent opportunity to allow a little architectural wisdom to take root organically. (I have found this to be the best way for wisdom to become permanent.) I decided to review Don's latest code, in the hopes of finding something that could help get my point across, and I came away with more than I had hoped. I double-checked my suspicions, and then picked up the phone.
"Hello, Jacqueline? How are you doing? Hey, do you guys still have that oddball statement cycle that comes out every Monday? Any others? Good. Listen, I'd like to ask a favor."
I placed a few more quick calls, and then went home for the weekend.
Monday went like most Mondays do: not fast enough. Finally, Monday evening arrived. I finished my day at work, and then went home, had dinner with my wife and kids, and went to bed early. I knew that the fireworks were coming.
The call came at about 11:30 P.M. It was Sam, the Ops manager at Jacqueline's data center. "Our totals are not footing for demand-deposit accruals! The job sheet says that we put in a new code patch; so, what gives?"
Sam had a legitimate complaint. The application over which he was watching provided totals at the end of every batch cycle that offered a summary of the previous run's ending balance, the debits and credits, and the ending total from the current run. If they did not balance (and tonight they did not), the operator knew that something was wrong. I told Sam that I would call him back, after I had talked it over with the guy who commissioned the change. I got Don on the phone, and then suggested that we drive in to the office where we could meet up and look for the bug.
"So, Don, what do you think is the deal?"
"I don't know. I wrote the specs, and then reviewed the code myself. I even ran it past some data from every data center, and everything checked out fine!"
"Everything? Did you foot all of the accrual totals?"
"Well, no. I did look at representative accounts of every type, however!"
"But didn't sum them all, before and after you ran the application for a day?"
"No, I'm afraid that I didn't think to do that."
"Let's look at the code, then."
A few minutes later, we had "found" the error, even though I had actually spotted the bug on the previous Friday. I let Don make the discovery. It was in his design and not the implementation. Then, I initiated the conversation on how to go about fixing the problem.
"I guess we will have to work out a fix, then call Woodgrove Bank and ask for them to replace the code that we shipped."
"Well, we could do that, but it might take them some time to work our code into their customizations. What about the statements that have been printed?"
Don's face fell. He had not considered that his oversight might involve tangible, cost-incurring wasted materials. I felt bad for Don at that point, but had to raise yet another point.
"Let's not forget about the other data centers, too."
Don's face was a comedy of emotions: panic, confusion, rapid churning, and then finally a strange sort of vindictive triumph.
"None of the others have called! They were not affected! Only one data center has to be patched; the bug must affect only some accounts!"
"Don, final accruals—where we actually add the interest that we have accrued to the customer's balance—do not happen until a statement cycle. I think that tonight we have only one statement cycle across the centers: Woodgrove. We need to issue a patch to all of the centers."
Don's face fell again. His goof-up would be made public to all of the data centers.
We called Jacqueline at about two in the morning. Sam had already called her, so she was already awake and primed for the conversation. I put her on the speakerphone, but did not reveal that Don was sitting beside me.
"Hey, Sam gave me the heads-up, so I took a look over the code already. I think I see what we need to do to fix it."
"Thanks, Jacqueline. Sorry we missed this one."
"Oh, that's alright. Was it the new guy?"
"Um, I think that he had a role in it, but there's plenty of blame to go around."
"Yeah, there usually is. It's just that he seems to think that he invented the digital computer! He gave Mark over at Haddleyburg a pretty good talking-down the other week."
Don squirmed in his seat, his face ashen. I put my finger to my lips to motion him to remain silent.
"I'm sorry about that. He's not a bad architect—just a little green. When I think back to my learning period, I can hardly stand the memories. They're so painful!"
Jacqueline laughed and then said, "Hey, look, I think I've got the fix for this code. Do you want me to test it, and then send it to the other guys?"
"Yeah, that'd be great! Can we have a copy, too, so that we can work it into the base?"
"Sure, but I'm warning you: My code won't have all of the fancy class interactions that ours did. It'll be pretty minimalist code, but it'll be easy to read and understand."
"Then, that's just what we need. Please work your magic, Jacqueline!"
"Will do. Oh, and: one more thing."
"You guys owe me a cheeseburger dinner."
"Of course! I'll even spring for the fries, this time!" The first time that she had helped me through an embarrassing goof-up, I had bought Jacqueline a cheeseburger dinner to thank her, and now it was a running joke between us. I made a mental note to tell Don about it later.
Jacqueline laughed and hung up. Her code was in my e-mail the next morning, and copied to Don. We tested it against the entire test data set, and, of course, it worked flawlessly.
Jacqueline arranged for a rerun of that night's processing cycle, so that the sole affected data center was corrected by the start of business the next day. The original statements had been spooled to disk, but not printed—a fact that I conveniently neglected to mention to Don. Better to let him suffer that false guilt, so that the lesson would stick. I knew that things were going to work out. By the next day, Don called to ask humbly if I could share some tips with him about architecting for maintenance tasks. We set up an appointment for the next afternoon.
"I sure learned a thing or two on Monday night. Can you please share your maintenance best practices with me?"
Keep Your Audience in Mind
"I'd say that some of the most important things to remember are to: keep your audience in mind, and the maintenance coder is your audience; maintain good working relationships with all of your stakeholders; and remember that Ops and maintenance can save your bacon, if they are inclined to want to help you."
"Boy, did I ever learn those last couple this week! But, what do you mean by 'the maintenance coder is my audience'"
"Every component that you design, every module that you architect, should be built with maintenance in mind. Software should be written to be read, not for absolute peak efficiency in all places; otherwise, we would be writing everything in assembly code. Things should be built in easy-to-read abstractions, so that a child could understand the relations between classes by telling interactions like a story. If you cannot describe your work in terms that a child can understand, you can probably stand to refactor things a bit, so that it becomes more understandable. You will be grateful later that you did it, because you will probably have to revisit your own code and will have forgotten all about the fancy little features that you cleverly built in."
I was on a roll, so I stayed on-topic a bit more. "Remember, premature optimization is the root of all evil. So, write your application in large-grained abstractions, and then later use a profiler to find the places in which you can tweak it for more speed. Avoid the temptation to apply fancy new doodads that you have found in magazines and books. Those have their place, too—but not until you have had a chance to study them in throwaway code situations a few times, so that you can become comfortable with the surroundings for those constructs."
"Are you saying that my designs should contain only plain-vanilla stuff? How will it be better than any other code?"
"You'll know code that you write or specify will be really, really good when your maintenance coders understand it as soon as they read it. Application architects sometimes get a big head, thinking about all of the clever features of their design; but, if it's not easy to read, the worth of the application is greatly diminished. To sum up this point: Remember that it's much, much easier to write code than it is to read it. That's why truly great architects are well respected by the maintenance coders. Lesser architects are well respected, usually, by only themselves."
Maintenance Is Where the Money Is
Don had the humility to blush a little, and then he asked a final question. "Is there anything else that I should know about the whole maintenance domain?"
"I guess there is one truth that will eventually reveal itself to you: Maintenance is where the money is, after the big development effort is over."
"What do you mean?"
"Big software-development efforts are always exciting, and it's great to work with green-field projects. But, sooner or later, if you're building things right, you'll be nearly done architecting and building that application. That's when the enhancements and corrections start—in other words, maintenance. Most applications have much longer maintenance cycles than they do development cycles; so, there's a good deal of job security and money to be made there. Who knows? You might even find yourself doing maintenance work on projects that you designed up front. It's not uncommon at all; so, be careful how you design it!"
I'm pleased to report that Don took all of this advice (and more, especially from Jacqueline) to heart. He has become a fine young architect, and seems to be doing well in all aspects of the job. In fact, the only annoying aspect of his personality that remains is his insistence on bringing me cheeseburgers every time that he comes for advice.
· When one is architecting solutions, why is it important to think about maintenance?
· How did "soft skills" affect the outcome of this story? Why are they important for an architect?
· Which elements of this story demonstrated the value of business, instead of technical knowledge? Which characters held this knowledge?
· Martin, Robert C. Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River, NJ: Prentice Hall, 2003.
Read all of the open-source code that you can lay your hands on. Make a note of what you find easy to read and what you find difficult. That is what the maintenance coder sees, too!
Batch cycle—For file-oriented applications, refers to a periodic, scheduled invocation of the application against a set of files that contain input transactions. It is usually contrasted with "online" processing, which happens in an ad-hoc basis against a single input transaction.
Demand deposit—A checking-account application.
Design pattern—A solution to a reoccurring problem in software design. Design patterns give architects a way to discuss software-design features easily and in a familiar way.
Footing accruals—Summing the amount of interest that the application has calculated is owed to each of the interest-bearing accounts, as an Operational tool to check for reasonable values. Many financial applications accrue interest daily, but they add the interest to the account balance only monthly.
Ops—The Operations team, responsible for executing an application and monitoring it to ensure that it runs properly. The Ops team can include:
· A manager, to coordinate efforts.
· An operator, to invoke scripts and monitor jobs.
· An analyst, to monitor totals to ensure that reasonable values are produced.
Premature optimization—An effort to maximize speed in the application, executed prematurely. Without benefit of a profiler, it is easy to make false assumptions about the location of performance bottlenecks, which can lead to suboptimal application-design constructs and convoluted code.
Profiler—A tool that can examine a running application and provide metrics about which operations are most costly in terms of time, I/O, and other resources.
About the author
Rick Wagner is a member of the International Association of Software Architects and a Sun Certified Enterprise Architect. Occasionally, he speaks at IT conferences. Rick has worked as a maintenance coder, developer, and architect—sometimes, simultaneously!
This article was published in Skyscrapr, an online resource provided by Microsoft. To learn more about architecture and the architectural perspective, please visit skyscrapr.net.