Collaborative Productivity Through Context
Summary: Provide better communication and more effective collaboration, both internally and with partners, through "context," a concept introduced by Maarten Mullender that focuses on using your organizational infrastructure to allow human interactions to be readily available for later reference, in an automated and formalized way. (25 printed pages)
I believe that humans, rather than computers, drive the business processes and that we have to provide them with optimal computer support, rather than provide computers with human support—which is the current state of affairs.
In this paper, I focus on human collaboration. Information workers are responsible for customer engagements, partner relationships, and for driving the business. These interactions are largely collaborative and not formalized; therefore, supporting them means bringing automation to these processes. Those organizations that can bring this support to their information workers will have better interactions both internally and with partners, and will thus be more successful.
To this end, I look at the problem space and I introduce a concept that I call "context." People work in context; they change context when they change activity, and they change context when they change focus. I will introduce a technical view on context as a way to bridge the gap between the human worker and the organizational infrastructure.
The concepts in this paper build on the ideas laid out in my article Razorbills: What and How of Service Consumption (which appears on both the Architecture Journal Web site and MSDN). I included a brief overview of those concepts from the "Context" chapter of that paper.
Whereas in the average organization, the processes in place completely handle 80 percent of the business activities, 80 percent of the effort, and thus of the cost, goes into other processes. Those other processes consist of unstructured, everyday processes, processes that have not been modeled, including the handling of exceptions that occur with modeled and automated processes.
Unlike their name would suggest, these unmodeled or ad hoc processes are not random; the people involved know very well what they are doing and how they are moving the process forward. There is no formal description: because it was impossible, because it was too hard, or simply because it was too large of an investment. Likewise, the documents are not unstructured; they describe things for which no formalism has been provided, that could not be captured by the LOB systems. These unmodeled processes, documents, and communication have structure; it is merely un-formalized. We can help the workers and we can improve the processes by providing structure, by discovering structure, and by capturing structure without formalizing the processes, the activities or the documents.
These unmodeled processes are typically collaborative in nature and they rely on human judgment and inter-human relations. The IT infrastructure typically supports the established processes well, but is scrambling to support these other, much costlier ones. Better support for these would lead to more efficient handling, better-informed decisions, more customer satisfaction, seized opportunities, avoided mishaps, and even better employee satisfaction—all of which add to the bottom line.
How can we begin to streamline these collaborative activities and reduce their costs? To do this we need to be able to handle not only structured content, which we do well, but also unstructured content, which we do not really do at all. Only when we get the workers to record their unmodeled activities and share their information can the rest of the organization—the other information workers, the supervisors, and management—reap the benefits of that additional information. We can start this value chain if we bring that effort directly to those people that execute the work, where they execute it, and give them immediate value while they are working on that execution.
I assert that workers will only record their unmodeled activities and share their information if:
- Workers can enter the information from where they execute the original activity without additional effort.
- Workers experience immediate, and lasting, benefit from entering such information into the system.
This means that improving personal productivity is key to gaining collaborative productivity. I further assert that the means of improving this productivity is through what I call context.
When Jon Tobey, who helps me make my documents more readable, reviewed my first draft, he added the subtitle "Grey Elephants in Denmark." He told me it reminded him of the old mentalist trick where the mind reader instructs the audience to pick a random number, then follow a lengthy math process and using the answer think of a country, animal and color, then informs them "But there are no grey elephants in Denmark!" (See for instance, http://www.csiro.au/helix/experiments/MindReading.shtml) On the surface, it seems impossible that so many people, following a seemingly randomizing path could arrive at the same answer. This works, and we are amazed, because what seems like random or chaotic -and therefore unknowable—thought processes actually have some order, and therefore predictability. The trick seems amazing, but when you see how it's done, it's very logical. The mentalist has simply discovered a particular model, based on our common frame of reference (that is, context). Likewise, by harnessing context, we can build tools that will be very powerful in organizing the unstructured work environment.
At first, tackling unstructured or unmodeled work seems like an impossible task, but I think that if we look at it a little more closely, like the mentalist, we will find we have everything we need to do it.
I propose context as a means to provide high-value integration between unstructured content as offered by Office and real time collaboration (RTC), structured content as offered by line-of-business (LOB) systems, and formal and informal processes and collaboration. I believe that the proposed way of providing context to the user makes it easy, almost seamless, to capture the information. That it brings immediate value during the execution of activities, greatly aids the execution of subsequent activities, and will give the organization a much better understanding of the process flow in general and can consequently be used to optimize those flows.
I see the context of an activity as:
- Everything that is captured about the activity.
- Plus everything related to what is captured.
- Ordered by current relevancy.
Making entering information easy and beneficial to users plays right to the heart of the problem space. It helps the information worker execute activities using Office, using LOB systems, and our platform to integrate these with both formal and informal work.
To this end, I want infrastructure to offer a set of contexts, such as activity context, document, collaboration and communication contexts.
I want to enable users to capture structure in their otherwise unmodeled documents, window environments, and collaborations and help them retain, share and communicate that structure. I want users to be able to capture anything they want in a context. This provides association to that context. The infrastructure can then use those associations to enhance the experience even further.
I want solution providers building point solutions that consist of:
- UI parts that have explicit, but local behavior.
- Metadata that:
- Ties those UI parts to entities and
- Details relationships with role specific relevance
- Composable workflows. (Both complete flows and portions that can be inserted on the fly.)
- Document templates and structured document stores.
Here is a typical scenario: Chris is the on-site office manager for Contoso, a design and construction company building an office building. He deals with the employees, customers, and suppliers. He orders supplies and takes care of any scheduling conflicts, shortages, rework or other problems that come up. In short, he wears many hats and often bounces from one context to another. At any given time, he may be dealing with an employee sick day, customer quality concerns, or supply issues. An ongoing concern is the white Italian tiles the architect specified. The original supplier cannot supply as many as the project needs in time, so Chris has been doing research and looking for alternate suppliers that meet the specifications. He also needs to share his largely unstructured activities with other people in the group. Chris has the specification document, the alternative tiles, the project plan, the contacts of the alternative suppliers, and the internal stakeholders on his desktop. When he switches to another activity, all of this is lost. When he returns to this activity, Chris has to find all this information again. When he wants to communicate to the project lead and the interior designer, he has to describe the alternatives and they have to look these up.
We could really help Chris and his coworkers if we could provide him with these abilities:
- Set context
- Switch context
- Restore context
- Share context
- Communicate context
This way, he can re-enter his previous workflow "midstream" he does not have to find the specification, look up the tiles, open the schedule, review the emails, and bring up the Web sites for the alternate suppliers.
Not only does context help Chris be more productive by organizing his information, storing and restoring the work or activity context, it also helps communicate context and organize the collaborative effort. It helps structure the collaboration and provides insight into the ongoing processes, and it provides insight into the past processes, so that he can improve them over time.
Logically, I define context to be a container of references (as explained in chapter 3.1) to entities in LOB systems. Each type of context has its own specific logic but all have common external behavior.
Using a semantic description of services, as described in Razorbills, we can capture entities directly related to the activity and offer relevant information using the modeled or derived relationships between these entities. The introduction of inference engines and further research will improve the notion of relevance even further.
Most of the normal business processes today are supported by IT, and one might say that, overall, it handles most processes completely. Here, we want to concentrate on these other ones, the human-driven processes, the workflows and the collaborations. We can support these other ones better than we do now.
Advancing Human-Driven Processes
Most business processes complete without additional intervention. When a manager notices that fulfillment of a specific order will be too late, he or she can try to expedite the order. The manager will pick up the phone or send some e-mails to organize and possibly coordinate the desired activities. The order process will not stop without that intervention, the order will still be fulfilled albeit too late. The manager, through this intervention, only tries to expedite it. This kind of intervention is human intensive and costly; however, it can have great business impact and increased customer satisfaction. Not all human-driven processes revolve around process intervention. Healthcare is a particularly striking example of an industry in which efficiency and better-informed decisions can make a huge difference on the customer's—that is, the patient's—well-being. Many processes in healthcare alternate between unmodeled collaborations and well-established prescribed activities. At every point, there are doctors, nurses, and patients, taking decisions on next steps.
At the heart of these human-driven processes, not surprisingly, is a human, an information worker, executing an activity. This worker will try to gain insight in order to make the right decision. He or she will search and browse the information in the systems, will read the documents, and listen to the person on the phone. Then, based on all of this information, the worker makes a decision. The decision may be to ask for further information, it may be to send an e-mail asking someone to take action, it may be an action in one of the systems, or it may be any combination of these and other actions. All of the information—the gathered information, the activities, and the actions—lead to more information. If we could capture this information, another person in the process or the same person working on another activity or continuing the same activity can then use that information. This is a cycle of gaining insight, decision, action, communication, and oversight, which provides more information, leading to more insight, and so on. This cycle feeds the individual collaboration and it provides information to the supervisor of multiple such collaborations. It allows the supervisor to see where there may be a need for action; for instance, by involving other people. It also provides insight into the process structure, which the organization then uses to decide on, act on, and communicate process improvement and organizational improvement.
This cycle will help at the individual level as well as higher up in the organization. The individual interactions with customers will be smoother and gain more insight into customer desires, perhaps changing requirements. Not only is it important to gain that insight and act on it, it is just as important to gain and act on it quickly. When we discover the need for this new product, how quickly can we make it happen and how quickly can we prepare our ecosystem to market and sell the product? Can we be faster than our competitors? Is our service better and faster at the same or even lower cost?
IT infrastructure today poorly supports the virtuous cycle of insight, decision, action, communication, and oversight, as shown in figure 1. Manual processes lead to mistakes and extra work because much of the information about the activities is lost; often the available information is overlooked either because it is too much work for the individual user to investigate or because it is too hard to find. Manual processes are implemented over and over and never improve for the same reason. There is clearly a need to address the issue. Over the next years, successful, agile companies will tie their business practice into their IT. How can we help them?
Figure 1. The virtuous circle of the business practice
Many activities in a business are related to business processes, but these business processes have not been or have only partially been formalized into automated processes and workflows. Much of the information in everyday business gets lost, is inaccessible, or is even unknown to the people that need it, because people do much of the work informally through conversations, using the telephone, e-mail, or documents.
Even in formalized business processes, unplanned events happen. In all these cases, where unplanned, unformalized, or unforeseen events happen, it is hard to keep track of what happened and to communicate these events with other people involved in the process. Yet, it is important to share the information, to allow others to be actively involved, and to allow them consider the information in their decisions and actions.
Organizations capture a relatively repeatable and predictable process in a plan or workflow. As plans change, they track progress on these plans and make amendments, whether things go better or worse than planned. Workflow systems try to address this problem. Most systems allow the definition of a plan and support tracking progress. However, not many systems are so easy to use that it is advisable to define a plan for individual processes. Not many workflow systems support the flexibility to change flows; that is, they do not make it easy to escalate an activity or to have the activity executed in a different way or by different people. Either they do not allow a change of plan or they do not offer the functionality to the individual user.
However, these special, non-repeatable, processes, together with the variations on standard processes, make up much of the effort in the workplace. When the order has been released for production and one of the suppliers sends us a delay notification, how is that handled? When the customer asks for the color catalog to decide on a change, we may continue with the production, but we had better wait with the paint job and with ordering the paint. How is that handled?
So realistically, many processes consist of structured, formal flows, and unstructured—or better, un-formalized or un-modeled—collaborations. When there is a problem in a structured process, the collaboration on finding a solution may be ad hoc in some respects and formal in others. When the solution is found and needs to be taken in production, the process becomes very formal again. For example, the exception process—the moment when, during the production of cars, the door does not fit—is extremely well thought through and very formal. This is not uncommon. In fact, this happens all the time. The development and introduction of a new product, a new marketing campaign, expediting an order, or just any collaborative process, moves back and forth between the structured and unstructured, between the formalized and the unformalized, between the modeled and the unmodeled.
A collaborative workflow system, or perhaps I should call it a case management system, should be prepared to:
- Support making and following a plan (define and execute a flow).
- Change or refine a plan on the fly (adapt a flow).
- Track planned and unplanned activities:
- Associate events to a flow or plan.
- History, insight, and oversight.
Some workflow systems address the first two, and CRM systems typically address the third. However, we need the combination of all three. Moreover, this should be easy to do. It should not require a tech-savvy person. A manager should be able to do this and any participant of the collaboration should, when allowed, be able to do all of this. This would lower the investment enough to make managing one-off collaborations attractive.
Yet, I would argue that all of this is still insufficient. At Microsoft, we use a CRM system to keep details of customer interactions, opportunities, and activities. However, users will receive an e-mail from a customer, respond to the e-mail, and not record this in the CRM system. The information is incomplete or even absent despite the fact that this information contributes to their employee bonuses. Why is this and why would other companies fare better? I believe the answer lies in these simple facts:
- Employees do not want to do double work. They do not want to respond to the e-mail and then go to a separate application, navigate to the correct customer, navigate to the correct opportunity, and then enter the e-mail information again. They want to take action from where they execute the original activity, in this case from the e-mail.
- The worker bees have no immediate benefit. The activity does not become easier, nor can they execute it faster. They can always do the additional work later, at the end of the month, and thus forget it so they never execute it or only execute it half-heartedly.
Reformulating these as requirements:
- Workers can enter the information from where they execute the original activity without additional effort.
- Associate from where the user is working.
- Association should not add overhead.
- Propose useful associations.
- Workers experience immediate, and lasting, benefit from entering such information into the system.
- Offer useful information.
- Navigate proposed associations.
- Navigate associated information.
- Store and mine the associations.
How do you find the information you need?
For me, it depends on what I need to find. When I need to find information on architecture and design principles on the Internet, I often do not know what is out there. I start with a search engine and from there on I will skim through the results and pick something promising. Then I typically browse from there or go back to the search results. When I have to find HR related information, I will visit the HR portal and skim the pages that I browse, following promising links. If it is something I have to visit more often, I will save it as a favorite or shortcut. I try to find something that provides me with links to where I would like to go and with information about those links. I do not typically use much structured information.
How would Chris, our purchasing agent, want to find information? Microsoft did research on this and we interviewed and observed purchase agents and other information workers. Our earlier research seemed to indicate that Chris and other information workers would take a document, skim through it, and read parts of it carefully. Then Chris would look at structured information, and use unstructured information: calls, e-mails, and anything else that helped move the purchasing process forward. However, when we provided Chris with new software that provided some history of the particular purchasing process and showed the e-mail or document content next to it, we found that Chris hardly looked at the document anymore. Chris primarily needed to read the document to find in what stage the process was, and with the new user experience this was immediately obvious. It also reduced the risk of misinterpretation, because in Chris's original approach the document did not always contain all relevant information to make the correct determination. Unfortunately, this software was only part of an experiment and has not been included in a product offering yet.
Finding, Not Searching
I do not want to enter information into a search engine and hope it will find relevant information. Searching is the second best alternative. I want the system to provide me with relevant information. I do not mind that there is some less relevant information as well, because I want to decide for myself what really is relevant. That means that I would like the system to search for me automatically, provide the right search criteria, and present the results. If the results do not satisfy me, I will use search as the next best alternative. I do not want the system to be pushy, but I do not want to be begging the system for information, either.
We can provide good search criteria while we are searching structured information. This is exactly what we did for Chris. We knew the relationships between purchase orders, products, and ongoing projects, which needed the materials. We could therefore collect the relevant information and present it. That is, we could do that as soon as Chris made the connection between the incoming e-mail and the correct purchasing process.
It will be harder to find or categorize unstructured information, or information that has structure that we have not modeled. It will be hard to provide the best search criteria for finding unstructured, uncategorized information, and it will also be hard to get people to help in categorizing that unstructured information. Documents themselves may be unstructured, but their store can be structured. Documents out on the Web are unstructured information kept in an unstructured store. So, search engines add their own structure on the top, for example, by counting the number of references to a certain page and the trustworthiness of the source of those references. File management systems such as SharePoint and document-centric systems such as Exchange already provide structure, and they provide properties that we can use. Once the properties are there, we can mine the information and push relevant information to the user. However, first we need the user's help to categorize the documents by setting those properties.
When Chris selects the e-mail, the system knows the sender's e-mail address and we can use that to find the contact information and the sender's organization. Both of these are important: the tile manufacturer, Fabrikam, provides links to the purchase order; the sender may work at Fabrikam and be responsible for arranging transport of the goods, or, it may be the person organizing those white tiles for Chris. The e-mail has a subject line and content, and we can parse both for certain keywords that give us further clues: a purchase order number, a product number, or a descriptive text such as "white tiles." Then we can mine the structured information to find matches, and we can even search in a role-dependent sequence. Inversely, the structured information will provide keywords that we can use for searching relevant documents. In the following section Context, I will describe a way to help us help the user. Clearly, this is an area where we can provide useful results by relatively simple means, but where we could provide much better results by applying results from MS-Research. For instance, our research indicates that there is a limited vocabulary used to indicate key entities. This set of words gains importance and then, after some time, loses importance again. Capturing this current vocabulary allows us to mine information and then link the information.
The Repetitive Nature of Finding
When Chris has collected all the information—about the Italian tiles, the project information, the e-mail exchange with the supplier and the project lead, and the alternatives to this specific tile—and made his phone calls, he starts on something else. He will not leave the desktop as is and do something else; he will need the desktop again. When he receives a response to his e-mail, he needs the entire context again.
This is a common problem. I know I found something yesterday, or last week, and now I need that information. I do not know where I found it and I can not remember where I left the link.
There is a need to save and restore context and there is an equal need to find context again, rather than having to find everything in that context separately. I would argue that by adding to the context, the user adds associations that may be helpful in other contexts, as well. Capturing these associations is an essential part of the virtuous cycle of insight, decision, action, communication, and oversight.
Chris should not have to search for all relevant information over and over again. During the collaboration, others should not be forced to search the same information again. Chris should be able to retain the context, he should be able to share context, and he should be able to communicate context. From there, Chris and the others in the collaboration should be able to navigate to related information; in other words, be able to find information rather than having to search for it (again).
People, in various environments, recognize the need for collaborative and activity-oriented solutions and are building these. These solutions, however, are often what I would call "point-solutions." They solve a specific problem or set of problems; other problems have to be solved separately. We may help Chris in his purchasing. We would like to help him with his other activities, and we would like to help his coworkers in their shared projects and in their other activities, as well. If we want to scale providing solutions, we have to look at the organizations that pay for these solutions.
Organizations are no longer prepared to make big investments only argued with often long-term ROI. They want low-risk small investments that lead to immediate payback, either through cost reduction or through increased revenue. Businesses need solutions to specific problems. Analyzing the business as a whole, before building solutions, is clearly not feasible. The analysis would take more time than the business needs to change, rendering the analysis useless. This requires businesses to do small investments with expected short-term ROI and with manageable risk. These so-called beachhead projects with small investment and short time to deliver can be approved without having to go to upper management and they pose an acceptable risk.
Building point-solutions has the advantages that the organization:
- Needs a relatively small investment for a calculable benefit.
- Can manage the risk that the project be over budget.
- Has a short response time; that is, the time between identifying the problem and delivering the solution is short.
If we look at some examples of such point-solutions like CRM systems in general and more specifically Microsoft's internal Customer Explorer, we see that they have a limited scope. They fall short, for instance, when opportunities and activities need to be associated to documents, e-mails, and appointments. They fall short when used by unanticipated roles. To tie these in, as well, requires more (and more specific) effort because the general supporting infrastructure is missing.
When my manager asked me whether I had entered my vacation in the vacation tool, I could not remember. I knew I entered the vacation in my calendar, but I could not remember whether I also went to our vacation tool on our HR portal. Why can't we integrate that tool with our calendar system so that I do not have to enter the same data multiple times?
Many organizations are prepared to invest in solutions supporting the sales process, the production process, or the product design process. Not many will be prepared to spend additional money on the integration of a vacation registration tool, because the benefit is not worth the development effort. Additionally, such solutions are brittle and create silos, so that each new implementation makes each subsequent implementation more difficult and more expensive.
To solve this:
We have to provide architecture that facilitates and encourages building cost-effective small solutions that combine easily.
A related problem is that businesses change, and thus the requirements for solutions change. Organizations rely heavily on developers to build and adapt solutions. Whenever a new solution is desired, or even whenever a solution needs to be modified to support the changing business requirements, developers are needed. This is a very expensive proposition. It is clear that developers may be required to build the core of the solution; however, should they be required for each and every tweak of the system? Many organizations would prefer the business people, or at least people working for the business side, make the final adaptations. They prefer these business analysts tune the solutions and help end-users personalize their solutions for optimal result.
We have to provide architecture that facilitates business analysts and users to enhance, modify, and personalize solutions.
What we need is architecture for a solutions fabric. Architecture in which many smaller solutions can be built, and in which many of these smaller solutions can be composed into larger solutions, and in which these larger solutions can easily be extended or recomposed into other solutions. Such architecture would support small investments and give them incremental, predictable, and durable value.
Before I start on a more technical perspective, let me provide a fundament. The concepts of describing the semantics of services and of describing user experience based on this semantic description can be found elsewhere, but are still not commonplace. So, I have included them here.
I include a summary of my article Razorbills: What and How of Service Consumption, because the concepts outlined in the following rely on this model, which is also used in the Information Bridge Framework (IBF).
Services expose business functionality to their consumers. Business analysts and business users often prefer to think in terms of the entities they use. They understand the relationships between these entities and the actions that can be taken on those entities.
Most services encapsulate entities. However concrete these entities were for the designers of the service, for the consumer of the service they are abstract. The consumer cannot touch them or manipulate them; the consumer does not know whether they are stored in a single table or in a set of files. The consumer, however, does understand what they represent: Customer, Order, Problem report, and so on. The entity is made up of the combination of all of its states and its behaviors, as well as its relations to other entities.
The service exposes views on these entities. The consumer may obtain an address view, a financial view, or a historical view on the customer. These views are snapshots of the entity, taken from a specific viewpoint. The views are concrete. They have a well-defined schema exposing attributes or properties of the entity. It must be noted, however, that views do not necessarily present a subset of the entity information. They may very well contain information from related entities and overlap with views on those entities. The view on an order may contain the customer's name and address as well as the product descriptions, so that it may be used in the user interface or used to populate a document.
The service also exposes methods or actions. Many of the actions are related to an entity. For example," release order" or "upgrade customer" are clearly related to specific entities. And, if these actions require input information, this information often comes from views on other entities. For example, a request to a supplier to create a sales order is built using the information from a purchase proposal. A view on the purchase proposal is converted into the sales order request.
Entities, views, and actions are artifacts that we can describe, that a user can understand, and that may help us describe services in ways that can be understood by business analysts.
I want to mention other artifacts that are less often used.
Many entities can be identified using key attributes. However, they can often also be uniquely identified using other attributes. Customers can be identified using their social security number, tax number, or DUNS number. A list of customers can be retrieved via the zip code, the region, or the account manager. We can formalize this through the concept of reference. A reference defines either a unique instance or a list of instances. A reference, like a view, has a schema.
The actions exposed by the service accept references to identify the entities on which they operate. The references are a more formal way of defining possible input as well as a more generic way.
References formulate queries to services. Such a query may be as simple as the list of key attributes or as complex as the formulation of a complete set of conditions. Often a reference will consist of a combination of required and optional attributes allowing clients that have different information to obtain the same views and use the same actions through the same interfaces. Design of reference schemas is just as important and just as complex as the design of view schemas.
Now, if we have a view of one entity, an order say, it should not be too hard to build a reference to a related entity—the customer, for example—for that order. In other words, we can transform the order-view into a customer-reference that refers to the customer entity. This effectively gives us a relationship between the order view and the customer entity.
A relationship uses the data in one view to construct a reference to another entity. Such a relationship can be defined using the schemas of a view and a reference. Consequently, relationships are not constrained to a single service; they can define relationships between entities encapsulated in different services.
If we are able to describe these entities, the views on these entities, the relationships between these entities, and the actions on these entities in terms that are understood both by the business people and the software, we would have a description of the "what." This description should not only describe the artifacts and the relation between them, it should also describe their behavior; what people can do with them, and what software can do with them. This description can easily be linked to, and in fact has to build on, the existing description of the "how."
Even though the designer of the service interface can provide much of the information similar to how WSDL is currently provided, an extension to WSDL cannot solve all of our needs. Relationships between entities offered by different services should not be stored in either, but would require storage external to these services. Instance specific information—such as the current availability of relationships and actions—as well as information created at run-time—such as tasks in a collaborative workflow—require run-time interrogation of the service and thus a run-time description. This is very much related to service contracts and the required infrastructure for providing run-time information about the state of a specific conversation.
Since views are what the designer exposes of the entities, much of the information is available in the systems today. Every service that was designed around entities provides most of the information needed, and in many other systems the analysis has been done and has been captured in some form of metadata.
I have to highlight the concept of a reference. An Order-ID can identify an order and a Customer-ID a customer. Most services implement methods that allow you to retrieve information by a key attribute like:
Customer GetCustomer(int CustomerID);
If I have an integer, like 4711, I don't know whether that represents a customer number or anything else. However, I can make that clear by wrapping it in some XML. For example, I can make it something like:
<Reference Entity="Customer"> <CustomerReference CustomerID="4711"/> </Reference>
I can then also change the method into something like:
Customer GetCustomer(CustomerReference CustomerRef);
I can keep the customer reference as simple as in the example or I can extend its definition so I can specify customers by their telephone number, their social security number, their DUNS number, or anything else that makes sense for the business. This would help us make the service interfaces more generally useable. Generic software could now access the information more easily and we could have metadata to describe the References, the information (Views) that can be obtained, the methods (Actions) that can be invoked, and the Relationships between them. This concept has been described in more detail in Razorbills. Above and beyond this, we can store references for later use, we can communicate about the entities using these references, and we cam have the other side understand what we're talking about. In IBF, references are stored in SmartTags in Office documents. Other software will need different methods of storing these references. In all cases, the software must be able to recognize it is dealing with a reference and it must understand what the subject of the reference is. In the previous example, I wrap the reference in an envelope so that the system can recognize the structure as a reference to the Customer entity.
Composite User Experience
Where Razorbills describes service semantics, it does not describe the solution's behavior. However, it does form a good basis for such descriptions; "descriptions," plural because many solutions can be built using the same set of services. Think of service aggregation or of the various user experiences that are possible: many devices as well as many roles and business strategies. Many LOB systems do not need to change when business changes. Financial accounting itself has not changed much, nor has order management; however, what has changed is how we use them. The process is always changing, but the LOB services change at a much lower rate. Therefore, I believe it is good practice to separate the solution description from the service description and spend more effort on making the service description reusable.
In the context of this paper, I want to focus on the presentation of information. For each view, or for all the views on an entity, we may define user experience. Such user experience may consist of viewing the information or of a combination of viewing and behavior. The view or entity has a local user experience: it has local behavior.
This behavior is, or at least can be, independent of the behavior of other views and entities. Metadata can either capture the complete behavior, or it can provide us with a description of where to find that behavior. So, either we may have generic, metadata-driven code, or we may have very specific code. In the latter case we may have metadata describing where to find that code, how to load it, and how to run it. In either case, given a reference to a view or entity, the solution metadata may provide us with the appropriate user experience.
Presentation and behavior can be different in different environments and for different roles. The presentation on my Smartphone is obviously more limited than it can be on my desktop, which is different yet from the presentation in the task pane of the Office applications. It is straightforward to extend metadata description to link different presentation and behavior to the service's entity description. This fits very nicely with the current trend towards composite applications, such as the one published on MSDN and gotdotnet.com: Smart Client—Composite UI Application Block. We can extend such a framework to support the additional smartness. The framework can find the metadata description to display the presentation for a given reference. It can find and offer the available related information and then it can navigate to (open) this related information. Not only can we find the presentation for any given reference, we can also go to any presentation part and ask what it is displaying; for example, customer Contoso with Customer-ID 4711. From there, the environment can offer the related information to the user, such as this customer's orders. In other words, we can associate Web parts, smart parts, UI parts, or whichever name we want to give these parts with the entity description and thus with a reference.
UI parts will not be the only type of rendering. I can imagine that given a Reference, the composite UI will provide me with a set of possible presentations. I can choose to display the reference as an icon with some pop-up information, as a piece of text, as a rectangle with information, or as a dialog or a set of dialogs. While displaying the reference as an icon, I may not need to retrieve a (or the) view on the entity from the service, but in most cases, if I want to display useful information, I will have to resolve the reference and fetch information. This information may be embedded in the reference, it may be cached in a service agent, or it may be retrieved on demand; our infrastructure and the UI part do not need to care. There should be a layer providing a Razorbills view on the information and that layer may be a metadata driven solution or not; we want to be independent of that choice. We want both our UI parts and our services to be independent of that choice. We want that to be an operational or situational decision and not a design decision.
Suppose Fabrikam and Chris's company agreed on a specific set of references; for instance, on references to orders (purchase orders for Chris and sales orders for Fabrikam), references to products, and perhaps one or two more. Then, Chris and Fabrikam could exchange information more easily. Chris could have an IM conversation (a real time collaboration session) with his counterpart at Fabrikam and exchange information about the purchase order and about alternative products. Each side would have their own interpretation of the references. Chris would look at a purchase order, the information obtained from Contoso's systems, and his discussion partner would see a sales order, coming from Fabrikam's systems. The UI parts would be different and the entities would be different. When they are looking at the products, the UI parts could be the same, as could the underlying information. Fabrikam might offer a Web service on the Internet and Chris's UI parts would show the information obtained from that site. So, the UI parts would offer the local user experience for a reference, based on the information that is available.
As a side note: clearly, there are UI parts for entities and views, but there may also be a need to provide user experience for relationships and actions.
I will use "reference" and "UI part" interchangeably. Although references are technically only the pointers to the service behavior and the parts define the user experience, one does not make sense without the other. The part together with the Composite UI framework is the behavior of a reference.
In the following sections, I will build on the semantic description of services and the composite user experience to provide a fabric for solutions that will help users take control over the processes and collaboration in which they participate.
Imagine that your computer would understand what you want to achieve and would help you accomplish that. Now, this may not even be a desirable goal since it would involve thought-reading and other scary techniques, but it would help if your computer had a better understanding of what it is you want to do. This would not even require magical powers on the side of the computer, because you give it enough hints to do a better job. You do open e-mails that have senders, recipients, subjects, and contents; you do open applications and navigate to business entities. The computer just has to understand these hints and do something with them. Not to constrain you in your activities and choices, but to help you by offering relevant information and relevant choices. It has to understand your context and the context of your activities, and then help you.
I use the "My Recent Documents" list on a regular basis, because it often has the document I want to open. In a very primitive way, it determines relevance as function of "recent." If my software truly knew the things that are relevant to me, it could always offer me useful favorites or preferred choices whenever I have to select or enter data. I could capture my unmodeled, but not unstructured, workflow including the projects in which I communicate, and therefore collaborate, with others. This would let me to work more efficiently and, what may be more, I would do things that I currently do not because they currently take too much time to find. Capturing and being able to use this information should make all of my work go easier.
The Worker's Context
Context is the environment in which something exists or happens. If we think about the context of something, we think about what is relevant to that something. Context is about relevance. Some things are more relevant than others; however, this relevance is not a static condition. Relevancy changes over time; some things become more and others less relevant. We also use the phrase "context change." Does our context change? Yes, our focus changes and therefore things that are very relevant suddenly become, temporarily, less relevant, whereas others become, temporarily, very relevant.
Chris, our purchasing agent, is reading an e-mail from his contact at Fabrikam notifying him that the white tiles have been delayed again, but should be ready soon. Our software tries to help Chris by analyzing the e-mail. Who sent the email? The system can relate the sender to a contact and then to a supplier. Are there any words in the e-mail we can relate to an existing activity? Then the system offers to restore the workspace or activity context Chris last used when looking at the purchase order, because that is the workspace that the software relates to Chris given the current context of the e-mail. There are more purchase orders with Fabrikam, but there is enough information in the e-mail to make this one the first choice.
Since we can teach the computer something about the relationships between suppliers, purchase orders, products, sales orders, and customers in general, we can support Chris in his purchasing. We understand the entities and relationships involved and we can infer what else may be relevant.
There are two important aspects, one explicit and one implicit:
- Context is about relevance to something.
- Context changes over time.
We need to look at context in relation to something now. We need to find what is relevant to the user now.
Let me relate this back to software and more specifically to the software used by information workers. The user's context would consist of his or her environment—everything in the user's environment. That is still not very practical. Let me narrow this down. The user's work context is everything related to the user's work. Not to the user's work in general, but to the user's work at this time. This will change while the user makes progress or when the user switches between activities. If I narrow this down even further, I may say that the user's work context is everything we can capture about what the user is doing now and about everything that we can relate to what we have captured. This is a reasonably flexible interpretation. In the future, with progressing technology and algorithms, we may be able to capture more and we may be able to relate more. However, context is also about relevance. Not everything we can capture is relevant. The main proposition of using context is finding and offering the most relevant parts of the context.
So, let me capture what I have so far:
The user's work context is:
- Everything we captured about the user's work,
- Plus everything related to what we captured,
- Ordered by current relevancy.
I can do the same for other contexts. The collaboration context is:
- Everything we captured about the collaboration,
- Plus everything related to what we captured,
- Ordered by current relevancy.
The activity context is:
- Everything we captured about the activity,
- Plus everything related to what we captured,
- Ordered by current relevancy.
I can do the same for the business context, the customer context, and so forth.
To keep this practical we can scope the captured information. The user's work context could capture information about active windows, windows in focus, the entities shown in those windows, and the information about phone calls. This information is not very relevant for the various people working collaboratively on a business process; they are more likely to be interested in the e-mail conversations, the contractual obligations, the status of the purchase order, and the contact information of suppliers and customer.
Let me give you an idea of what I think we can do with such context. For each type of context, we could have a specific implementation and a specific user experience:
- For the collaboration context, I imagine that we display a timeline, or a set of timelines, for things that happened and a plan looking forward. When Chris selects the e-mail informing him about the delayed tiles, the system offers, in a non-intrusive way, a set of relevant contexts. When Chris selects the proper collaboration context, the system automatically adds the information about the e-mail to the timeline using the time of sending. Chris can then mark the e-mail as a notification of delay. The other people involved in the collaboration would share the collaboration context. When the project lead looks at the collaboration context, the delay notification would be clearly visible.
- For the activity context, I imagine that Chris adds the entities of interest to a workspace. The e-mail in question could be on the workspace, the supplier information, the purchase order, and so on. Chris right-clicks on any of these and the activity context UI pops up the set of most relevant related instances, so that they can be added to the context, as well. The activity context provides an optimal user experience for uncovering information and for storing and restoring context for Chris. Normally, the user's activity context is not shared with others.
- The UI parts may respond to changes in context. I imagine that the contact list adapts to the current context and shows the contacts involved in the project, as well as the supplier's contacts.
I can drag-and-drop items from the activity context to the collaboration context and vice versa. Each would take the information and handle it in its own specific way. Each would present the information in its own way. That said, they have in common that they both accept new stuff of interest. Their internal logic and user experience is specific, but their external behavior is very similar.
This helps us capture much of the important information we need to share with others, and that may be needed to trigger different responses than the normal flow would. I see two elements that contribute to this:
- The infrastructure itself can capture actions that the user selected on the entities. The elements on the workspace interact with the context and the actions are executed in concert with that context.
- The user has an easy way of connecting items: static items such as contacts and products, as well as events such as e-mails and phone conversations to activities, collaborations, and other contexts that allow sharing of the information. In doing so, the user is given information about related facts and activities, and automatically has access to information found earlier.
Containers of References
This is so important, I will say it again. The activity context is:
- Everything we captured about the activity,
- Plus everything related to what we captured,
- Ordered by current relevancy.
I work on an activity and I create my context around it. I start working on a different activity, but I know that I need to continue on the current one later. I want to change context, but I want to restore the current context later.
With this in mind, we want to capture information, we want to store that information, and we want to get back to it later. We want to find related information and we want to hold onto it, too.
I have been a little loose with my terms here. A crucial question is whether we capture information or references to information. The difference is that locally kept (copies of) information may become invalid, whereas references to information allow us to retrieve the latest information. We do not always need to or even want to store all information locally. We may want to keep references to it and we may introduce service agents to manage access to that information, and we may have the service agent decide whether to maintain local copies. The Razorbills concept allows us to use references between applications: to store them in documents, e-mails, or in contexts. Note that often the information (a view) is a reference. This is the case if the view has the data needed to identify the underlying entity, such as an order number for the tiles.
I define context to be a container of references. Each type of context has its own specific logic but all have common external behavior.
For each of these I can imagine multiple designs or implementations based on different needs and different personal preferences. For instance, I imagine that:
- The activity context displays the references as icons on a workspace and the relationships as connecting lines. When I hover over the icons with my cursor, a popup displays; when I click, the popup text replaces the icon; and when I double-click, a window opens for the content.
- The collaboration context manages and shows timelines and planning. It shows the references, where appropriate, attached to the timeline or to the planning. It may display the references as icons, as pop-ups, or as windows. The references will exhibit the same behavior as in the activity context.
- A document context shows references embedded in the text as SmartTags or icons. These icons provide exact, up-to-date references for the text. In Office 12 or 13, these references will have behavior that is more similar to the one in previous two examples.
- A real time communication (RTC) context window shows contact information, video, instant message conversation text, possibly a transcript of a voice conversation, and an area that can be used to share (or actually communicate) references between the parties in conference. Two parties can view different information based on the same reference at the same time. They share the references, but not necessarily the behavior. (My purchase order may be your sales order.)
These contexts are similar in that they store references and interact with these references and allow these references to provide the same user experience across contexts. They are similar in that one may add references to them and remove references from them as well. They are also similar in that I can query them for a list of most relevant references.
They are different in what they do with the references, in how they structure these references, in how they present this structure, and in how they store and share the context. They are also different in how they determine relevance. The collaboration context may find recent events more relevant and it may understand something about the business objective. A document context has no timeline and no business objective, but it may derive information from its store or from its type and use this instead.
These contexts work together and users can drag and drop or copy and paste information between them. Each reference or UI part presents a set of relationships allowing the user to browse to other entities and thus to more information. However, the challenge is that there may be many relationships and that many of those will lead to huge result-lists. There may be many relationships from supplier to purchase orders, to contacts, to invoices, to general ledger accounts, to warning letters and then there may be refinements like from supplier to all purchase orders, to delayed purchase orders, to urgent purchase orders and so on.
Not only are there many relationships, but sometimes the result set is too large to be practical. A pull down menu with relationships should offer the five to ten most relevant relationships and those relationships should be defined such that the result set is practical. It should not take away access to the other relationships, but it should provide easy access to the most relevant ones. The context provides this relevancy. The context should facilitate access to all information, but volunteer the most relevant information first. If the activity context has a supplier, Fabrikam, and a product, the white tile in question, it may offer all purchase orders to Fabrikam, containing a white tile, and most likely, there is only one of those.
With this definition, context can be many things, as long as it exposes certain behavior. The purchase order can be context as long as it exposes the right behavior. It does not matter whether we model the purchase order as a document or as an entity in a purchasing service; I can ask it for products and receive the list of products on the purchase order or, depending on the intelligence of the context, I may receive those together with a set of alternative products. My activity context can refer to a project context, which refers to a purchase order, which refers to a supplier and products. If I were to ask both the activity context and the purchase order for a list of products, I would probably get different responses. Each has its own notion of relevance.
I discussed earlier that when Chris received the e-mail about the delayed order of tiles, our software could recognize a number of clues and provide context to the e-mail. Inversely, the e-mail should also become part of that context. Chris notifies the project manager via e-mail, asking whether he should look for alternative vendors. On receiving the response, the system should restore the activity context and the original e-mail should now be part of that context. From this, it may be clear that:
Context may contain references to local information. This information may consist of e-mails, local documents, notes, or anything else that is stored on the local system.
References, Context, and Collaboration
An interesting side step here is that Chris may want to add the e-mail as well as the rest of the e-mail conversation to the project context, so that the other project members can follow developments. The system must then make the e-mail conversation available, for instance in a SharePoint folder, and then refer to the shared version. In general, systems can only work with structured information; that is, with information that is stored in a structured way. As soon as documents, notes, or e-mails are stored in a structured store, the software can use them and we can define references to them.
The project members share the context that is relevant to the project. This project context is stored somewhere and the participants may want to communicate about the project and thus refer to the project and the project context. Project context, activity context, and in fact most contexts, can be stored somewhere, and thus references to these contexts can be made.
Context is an entity; references to context exist and context can thus be part of other contexts. For example, the activity context may contain one or more references to project context.
The proposed concept of context adds a layer on top of the services. This layer does not simply provide a view on those entities. It groups them, orders them, and considers their relevance. In other words, it provides context.
This context layer provides us with a web of relationships between those entities and it provides us with common behavior of those entities across devices, roles, and activities. This layer helps gain insight and, over time, this layer will provide more intelligence in providing the relevant relationships, the relevant entities and, in general, the relevant information. This layer helps provide oversight and, eventually, will be better at making it easier and more natural to associate events and entities.
The idea of an intelligent container of information, working with services, managing the interactions with these services and their user experience, and understanding relevance, is more important than the precise definition of the individual containers even though we will need those to do actual work because they offer the user experience. The UE is no longer the representation of a particular program; it becomes a building block in the collaborative flow.
In my opinion, the important parts are communicating information and capturing information. They offer the business concrete value in process effectiveness. The communication contexts and collaboration contexts offer this functionality. Context is important because it provides an easy way for people to organize information and thus provide the system with information about structure, associations, and, derived from that, relationships. The activity context is a personal way of offering structure and is a useful additional offering to capture structure that is not directly shared. Its main relevance is to lure the user into entering information we may capture and use to discover relevant associations to provide better communication and more effective collaboration.
Building the basic infrastructure will not be the challenge; we can do this using existing technology. I see two main challenges:
- The first challenge is to get to mutually understood schemas and key structures between organizations. I think this will be an evolutionary process. Much like UDDI was first used within organizations, we should not expect service semantics to be exposed for general public consumption immediately. Instead, we should expect organizations to build internal solutions that they tie together. Then, at some point, they may expose these solutions to trusted partners. These partners would only have to agree on a very limited set of references and views; only on products and orders, for example.
- The second challenge is in discovering the structure in the unformalized work. The person dealing with unmodeled information, the "unstructured" information worker if you will (pun intended), does not really deal with unstructured information. He or she deals with information that has not been formally analyzed and modeled, but the structure exists in the heads of the people—it is just not formally captured, and thus the supporting software does not understand it. The activity context as well as the other contexts may be places where people organize their information implicitly or explicitly, and thus provide us with an opportunity of capturing, at least part of, that structure. This is an area where additional research will undoubtedly provide a lot of value. Currently much of this research aims at analyzing the few elements that have been structured and that can be analyzed. The Razorbills description of services and the use of references in the context layer may add new sources of information and may aid the research into inferring relevance. It's a bit of a chicken-and-egg problem: the inference engines are not as useful as they could be without the semantic description of services, and the semantic description of services is not as interesting as it could without good inference engines.
Building such infrastructure today, with our understanding of entities and their relationships, will be great help for the worker dealing with structured information. Gradual improvement of that infrastructure with inference engines and a broader understanding of service semantics and document structure will gradually improve the usefulness and extend its usefulness into new areas.
If you, like me, believe that humans, rather than computers, drive business processes, and that we have to provide them with optimal computer support rather than force the humans to adapt to the systems, we should think what that means for the software we offer them.
The ideas presented in this paper combine a number of ideas that I believe are very important—that must, and eventually will, happen—and a number of ideas that could be shaped in alternative ways.
I believe we need a semantic description of services. We need a description of what services offer so that generic software may use these services and offer their functionality to the users.
I also strongly believe that we need to capture more of what the user is doing and try to infer what the user needs or would like. We need to provide the user with more relevant information and more relevant choices.
In "La Fontana," a Sicilian restaurant in Seattle, I noticed a quote from Leonardo da Vinci painted on the wall:
L'Architecturra é semplice
Ma la semplicitá é difficile
I translated this into:
Architecture is simple
But, simplicity is hard
We can often find complex solutions for complex problems. However, whenever the problem changes slightly, we have to make major adjustments to the solution. It is hard and time-consuming to find simple yet sufficient solutions for complex problems. However, they are worth the effort, since they will be more resilient to change.
I think the idea of context presented in this document is simple and powerful. I believe it is a good way to start making our software human-friendly, and holds the promise of gradual improvement. We can replace individual components with better, more intelligent ones without affecting the others. This means we can insert better ways of defining relevance into the solutions as they become available. We can start building small useful solutions and we can combine these smaller solutions into bigger ones as more solutions become available. We can change the composition of these larger solutions and replace the smaller ones as business changes.
I believe my definition of context fits very nicely with the growing interest in composite applications. The contexts, the containers of references, do not need to understand the contained references. They merely need to interact with them and with the UI parts that provide their user experience. They need to understand where to find that user experience, which modeled relationships there are, and add relationships or associations by merely allowing the user to add references to the context. I believe the semantic description of services used by smart composite applications and supported by context together provide a solution fabric as described earlier. As descriptions of new entities are added, their references are published, and UI parts are made available, the reach of the solutions grows.
Human work is not only one of the bigger costs in an organization; it is also one of the bigger opportunities. Improving the effectiveness of peoples' work, reducing the repetitive and error-prone elements, helping them organize the things they work with, and providing them with relevant information, should lead to improved quality of their work and to more satisfaction in their work. This, in turn, should lead to seized opportunities that either have an immediate effect on cost or revenue or have a longer time effect on customer and partner relationships. It is an investment in the human capital of the company.
About the author
Maarten Mullenderworks as an architect for the Office Business Applications department at Microsoft. This group builds applications and infrastructure to bridge the structured world of traditional business applications with the un-modeled world of Office Applications. Before this, Maarten worked as a solutions architect for the .NET Enterprise Architecture Team on architectural guidance to enterprise architects.
During the last eight years, Maarten has been working at Microsoft with a select group of enterprise partners and customers on projects that were exploring the boundaries of Microsoft's product offerings. The main objectives of these projects were to solve a real customer need and, in doing so, improve Microsoft's products.
Maarten has been working for over twenty-five years as a program manager, as a product manager, and as an architect on a wide range of products.