Ray Elenteny

November 2007

Revised February 2008

Summary: Supporting an application begins before the first line of code is written. (6 printed pages)

Contents

Introduction
Support Starts at the Beginning
People Are the Heart of Support
Tools Are Important, Too
Conclusion

Introduction

I stopped by the customer-support manager's office. "Did we get the CD of log files from the customer?" I asked.

"Yes, we did." He fidgeted under my inquisitive gaze and added, "But not all of the log files that we need are on it."

"Great," I sighed. "How can we solve this problem under the time crunch we're in?" My question was not directed at anyone in particular. We had to get this problem solved quickly.

"We're asking the customer to send smaller chunks of the log files through e-mail," the customer-support manager said. "We can't get access to their system, so we've got some screen shots coming, too. It's going to take a while."

I had just left a meeting in which it was made very clear that we were trying to close a deal with this customer, and that it meant a lot to the company. Now, they had a problem, and I knew that it was going to be very difficult for us to clear up this problem quickly. There was not enough data to make a decision, let alone a decent guess, and we could not get to the data that we needed. It was not a good day to be in the office.

It was the last week of our fiscal quarter. It was always hectic—an all-hands-on-deck affair. In development, we were made aware of a handful of deals that were in the process of being closed. It did not matter what your title was; you could be called into just about any situation. This was a pretty typical pattern for us. The trick, as always, was in trying to keep our projects moving ahead during the inevitable interruptions.

The end of a quarter was always a give-and-take situation, especially with existing customers looking to purchase additional products, renew maintenance, or contract for services work. Occasionally, in the process of posturing for a good deal, customers liked to point out the problems that they have had with our software. It was always rough on the development team to hear about these problems from sales. Our software was hindering them from closing a deal. It did not happen a lot, but that did not make it any less difficult to handle. Thick skin and a smile were required. It was just part of the business.

This quarter was particularly memorable. One of our largest customers was looking to expand the use of our products. Several of us were on "alert," waiting to assist in any way that we could to help close this deal. Then, the call came. The customer was having problems with our application. This was not the call that we were expecting. Customer support was already in the process of trying to diagnose the problem; they were working through their standard troubleshooting procedures. Unfortunately, this took quite some time, because log files had to be gathered from several locations. This customer's installation was very large. Even "zipped," the gathered log files were many megabytes in size—much too large for e-mail. The only way to get these files to our site was to have them placed on a CD and sent via overnight carrier. We had yet to implement a mechanism, such as an FTP site, to transfer large files. The customer was able to get the log files on the CD and ship them overnight, with little time to spare. Knowing the time frame in which we were working, we all felt the pressure.

The CD arrived the next morning. We quickly "unzipped" its contents. Some of the log files that we needed were not included on the CD! At about this time, the salesperson who was trying to close this deal stopped by and asked, "How's it going?"

"Fine," we replied, with the most optimistic look that we could put on our faces. From previous experiences with this customer, we knew that we could not get access to their system through a VPN or secure shell connection. The network operations people just did not allow it.

The Vice President of Sales soon stopped by, and gave us a little pep talk: "This is a nice-size deal for us. We know you guys are doing your best. Let me know if there's anything I can do." Translating the sales speak, this meant that we would lose this deal if this problem did not get solved very quickly.

We got back on the telephone with the customer to start working through the missing log files. We asked them to send smaller chunks of the logs via e-mail, in an attempt to determine the problem. It was painful, to say the least. "Do you have access to any Internet meeting software?" we asked the customer.

They replied, "No. Do you?" We did not. This could have been very helpful, because we could have gotten access to their site in a safe manner through a third party.

My phone rang. I looked at the caller ID; it was the CEO. I picked up the phone. "I thought I'd check in to see how things are going," he said.

Now, awareness of the problem had reached the top. "We're working through it," I said. "I know the team will make this happen for the company," was his response. "Keep up the good work."

In reality, we were blind to what was going on with the customer's system. There were too many log files to disseminate. We could not get any real data back and forth between us. Asking the customer to type various commands and read the response had become mind-numbing. E-mailing screen shots from a system is a far cry from getting your hands on the system. We all knew that we were in for a long night.

Although this is a fictional portrayal of a customer-support nightmare, many who have dealt in customer support can relate to it. Here, the story touches on difficulties that arise in supporting a customer from a software vendor's perspective. Those who develop internal applications would quickly be able to share their user-support war stories, and those stories would be just as messy.

Support Starts at the Beginning

After an application has been released and installed, it is into the midst of customer or user support that the pressures of delivery are transferred. In some respects, the inability to address support issues in a timely and efficient manner has greater consequence than a missed development deadline. It is often visible to a wider number of people. End users can be significantly affected. Quite frankly, even though grumblings abound, missing delivery deadlines has become an unfortunate stigma in software development. Many people just expect it to happen. Conversely, extended time periods in resolving support issues are not tolerated.

Software support must start at the inception of a project. That might sound a bit exaggerated, but think about it for a minute. How long after the coding effort begins does the first debugging or logging line of code get written? I would not be going too far out on a limb to say that it's probably within the first hour—during the first day, at the absolute latest. If the concept of supporting the software in development has not been discussed before the coding has begun, what is going to happen to the logging code that is being written on the first day? Hopefully, the developers start asking questions, and do not start making assumptions.

Determining how to best plan this development effort requires input from many sources. Input must be gathered, and then reviewed; and requirements must be determined from two perspectives. The first perspective is historical. A team must be assembled that will provide a great deal of information with regard to how the support of an application has been handled in the past—including what has worked, what has not worked, and what is missing.

Even with a new application, the team should be able to provide valuable input, based upon past experience. The team will also have to look ahead. What are the new or different aspects of this project that might introduce a new supporting requirement? Are there new tools available that assist in resolving problems more quickly? How will the new application components tie into the existing support process? In assembling a team, consider drawing upon the knowledge and experience from the following areas within your organization.

User/Customer Support

These are the people in the trenches who support the users. In my opinion, there is no single group within an organization that has a better perspective on how well the current support architecture is performing. These people tend also to have a great deal of input on how to improve the support of the application.

Developers

When the support team needs help, they will turn to the developers. The developers might have to dig deeper than the support team. Because they have a deeper knowledge of how the software is written, they will also have a different perspective in terms of examining the application.

Consultants

Enlist consultants who are familiar with the software, or enlist the services of other partners; they will contribute yet another technical angle.

Users/Customers

Often, users are asked to gather information that is used in supporting an application. They will be able to share with the team the level of difficulty, or how much trouble it takes to gather the information.

Data Access Is Paramount

Armed with input from the team, how is all of this information applied? The bottom line to most support needs is access to data. Whether the data is made available for human consumption or for a more sophisticated automated system, the old adage of "garbage in, garbage out" applies. Support data comes in various forms. The following points call out some common support data and its uses.

Debug Logs

This is about as basic as it gets. Logs might be written to files, or sent to databases or any number of other destinations. Data that is written to them should be accurate, relevant, and concise. As basic as they are, I cannot overstate the importance of even this foundational aspect of software support. Especially with software logging; after a logging system is in place, it is rarely ever improved and corrected. Logs will be reviewed in the future, and the team will say, "Boy, we really need to fix the logging in that module." However, unless those improvements are scheduled for enhancement or significant refactoring, they will never happen.

Event Logs

Event logs are a bit different from debug logs. Typically, they will provide data in a more uniform fashion than a debug log. They are often stored as records within a database, to be viewed later with a reporting tool. For example, where a debug log might provide the details of the process that is used in printing a document, an event log would likely submit a single record that states that a document was printed.

Logging Levels

The amount of data that is sent to logs does have to be controlled. There are times in which every detail that is going on in a system must be known. Most of the time, just having to know when something goes wrong is all that is necessary. You must determine a policy for setting the conditions for when data is sent to log files. The levels should be easily configurable by anyone who has administrative access to the application.

Installation Support

If the application is to be installed multiple times, it is imperative that tremendous amounts of detail with regard to the installation process somehow be available. It is almost impossible to go overboard in tracking the progress of an application's installation. There are too many things that can go wrong, and too many variables that can affect an application's installation. If it seems that too much information is being gathered during the installation process, it is probably close to the amount of information that must be tracked!

Monitors

These are separate processes or threads that either run as a part of an application or tap into the application. Monitors tend to collect data from multiple inputs, and provide near-real-time information on the state of an application. This information might be delivered through a simple e-mail interface or a graphical diagnostic dashboard. Existing types of monitors range from simple, homegrown processes to large, enterprise-wide commercial packages.

Reports

Reports can be used to present different views of a system. Administrative reports can provide information with regard to the configuration of an application. Reports can be provided that parse through log files or database records, to provide any number of views on the stored data.

People Are the Heart of Support

Understand that, just as important as the help that the architecture might provide in supporting software, application support is a process that involves people; it is a hands-on personal experience. Given this fact, the best support is provided when good relationships exist within and across organizational boundaries—from the development group, through user support, and, ultimately, to the user. There are times in which application support can be frustrating. Everyone who is involved wants to see a problem solved, but the solution is just not apparent. Expect this. It will happen. Realize that everyone who is involved wants the problem to be solved. Finger-pointing cannot be allowed. Although the relationship is a bit different with external customers, my experience has shown me that they would rather work as teammates than as adversaries when problems arise. Getting to the solution is a winning proposition for all who are involved. Any development team should be ready and willing to take part in problem-solving.

Tools Are Important, Too

Whether within the same company or not, communication might not be as easy as walking down the hall and having a conversation. Whether an organization is large or small, it is wise to invest in a few key tools that will help the entire support process. What these tools are might be different, based on the type of support that is required for an application. Listen to the various people who are working to solve a problem. When someone says, "Boy, it would be nice if...," think about what has just been asked for. Does it make sense? Would it really help the support effort? Many times, it will.

As seen in our introductory scenario, things as simple as an FTP site for transferring files are invaluable tools. A Web site that contains how-to information or knowledgebase articles helps support efforts significantly. Linking a Customer-Relationship Management (CRM) system to a development-issue tracking system provides relevant data and a communications link within an organization. These tools do not have to be overly sophisticated or fancy. They just need to enable the flow of data and communication efficiently in the application-support process.

Conclusion

When it comes to supporting software, the keyword is visibility. Applications should be architected to provide important data that is easily accessible and interpreted. It is important that data get into the hands of those who need it to solve a problem in an interpretable format. Keep an open line of communication with all who are involved, so that everyone can be kept up-to-date. In the majority of cases, the more eyes that there are looking at a problem, the more likely that the problem will be solved quickly. This pattern will not guarantee that a tough problem will never arise; but it will ensure that the right data, tools, and people are in place to address a problem in the best possible way.

About the author

For over 20 years, Ray Elenteny has worked primarily on server-side, cross-platform infrastructure development for software products. He thoroughly enjoys the rapid pace of the software industry. Applying relevant technology to business problems, interacting at all levels within an organization, and mentoring are some of the day-to-day activities in which he takes pleasure. Ray participates in Atlanta's IASA Chapter and Java User's Group as both attendee and speaker.

 

This article was published in Skyscrapr, an online resource provided by Microsoft. To learn more about architecture and the architectural perspective, please visit skyscrapr.net.