Enterprise Identity Synchronization Architecture

by Mike Morley and Barry Lawrence

Summary: With an increasingly diverse mix of hosted, product-based, and custom solutions found in the modern corporate environment, reliably ascertaining the identity of employees across the enterprise can be difficult, and is often left to manual processes.

Contents

Introduction
Scenario
Goals
Solution Approach
System Architecture
Active Directory/Exchange
Conclusion

Introduction

This article will discuss a system and approach that accomplishes identity synchronization, developed by BST Global, a Microsoft Gold Partner, and deployed at Matrix Solutions. The discussion will focus on the first phase of the project describing the architecture of the Employee Identity Synchronization system, which synchronizes employee identity from Ceridian’s hosted HR Payroll, BST Global’s Enterprise PSA, Microsoft’s Active Directory, and Exchange Server.

Each individual system contains domain-specific data regarding the employee. These systems also use their own terminology to describe the data—terminology that may not be applicable to the overall corporate understanding of those data, or relevant to other system domains. HR/Payroll systems, for example, typically contain extensive information that only HR requires in order to produce payroll and manage the employee, whereas PSA (Professional Service Automation) systems contain employee data that relate to projects, employee charge rates as applied to projects, time, and expense data.

To synchronize effectively the relevant data across systems, a specification that defines the employee data in terminology relevant to the corporation, regardless of system domains, is required. To address the challenge, Matrix Solutions Incorporated (Matrix) has adopted the use of a SOA engine called the BST Freedom Exchange (BSTFX), a services-based workflow engine that facilitates the corporate definition and synchronization of federated keys and employee identity across systems. The workflow engine has the ability to translate data from system domains to the corporate domain and handles the process of populating the corporate definition from the individual systems, managing changes in those data, and synchronizing the resulting data back into the subscribing systems.

Scenario

Matrix Solutions Inc. is a rapidly growing Environmental Engineering consulting firm based in Calgary (Alberta, Canada) that primarily serves the upstream oil and gas industry. The company’s rapid growth from 100 to 250 employees in four years makes managing employee identity a challenge.

As with many modern medium-sized firms, Matrix makes use of a ‘best of breed’ approach to primary business systems rather than attempting to use one monolithic ERP (Enterprise Resource Planning) system to manage core business information. While this approach increases efficacy of each system within a particular business domain, the fact that each system will have its own native database and associated identity management framework exacerbates the issue of identity management.

Matrix uses a mix of commercial hosted onsite, commercial hosted offsite and custom developed systems to run its business. Each has its own database and domain-specific method for storing and describing business data.

Identity is at the heart of every system, as it defines the user experience within each domain, and is the means through which security is governed. Security is dependent on proper knowledge of not only identity for authentication but also for authority and access to the correct information within the system in question. Keeping employee identity current in all core systems is therefore critical to the smooth and secure operation of the business.

Synchronization of identity between systems at Matrix traditionally has been addressed by manual methods using full-time staff responsible for each system. In some cases, batch-data reconciliations between systems were conducted by temporary staff to bring systems back into sync, resulting in identity disconnects between systems and data inconsistency. One specific example of this involved an employee who had returned from an extended leave having two separate identity accounts created for them. Because of the manual methods that are used to synchronize identity, it was not until sometime later that the error was detected, creating a great deal of extraneous work in reconciling and resynchronizing the systems that used the information.

Matrix operates in the dynamic, fast moving, and cost-competitive consulting business sector. The consulting business model is particularly sensitive to overhead costs—each additional overhead resource required to support the business has an impact on profitability. Keeping the number of administrative resources at an optimal level to support the operation of the business while ensuring they are being used to maximum effect is crucial to the success of the company. Having administrators dedicated to keeping systems synchronized by doing data-entry is not in the best interest of the organization or the resource.

Goals

Based on the business challenges, the following key objectives were established for a key IT initiative:

·         Reduce the friction of information flow throughout the organization.

·          Maximize use of knowledge and administrative assets, wherever possible.

·         Replace repetitive manual business processes with automated processes.

·         Get the correct information to people when they need it, in the form in which they need it.

In addition, the following architectural guidelines were established to govern the technical solution:

·         Create a set of standard definitions that describes core business objects in terminology Matrix understands (domain-specific language).

·         Create an extensible integration framework with the ability to capture critical information from many sources and build a cohesive view of the current state of these Matrix Business Objects.

·         Use the Matrix Business Objects to keep subscribing business systems synchronized and up to date using a rules-based workflow engine that can route and handle exceptions.

·         Construct a standard set of services using the Matrix Business Object definitions that can be reused for other business purposes.

Solution Approach

Matrix engaged with BST for consulting expertise and best practice guidance on systems integration. The project team consisted of Mike Morley and Adrian Brudnicki from Matrix Solutions, and David Hebbler from BST, working over the course of a six-week Enablement Program that was established to provide Matrix with a hands-on, guided approach leveraging the BSTFX platform:

·         Week 1: System design, architecture, and hands-on coaching with the Freedom Exchange integration platform offsite at BST

·         Week 2: Codevelopment and guided implementation onsite at Matrix to establish baseline interfaces and system classes

·         Weeks 3–5: Independent programming by the Matrix development team to flesh out the working details of the solution

·         Week 6: Onsite solution review, QA testing, and deployment readiness

System Architecture

Rather than a single monolithic system to manage core business data, Matrix uses best-of-breed systems for each business domain. To limit scope, three key systems were selected for the first release:

·         Ceridian’s HR/Payroll Web system

·         BST Global’s Professional Services Automation (PSA) System

·         Active Directory/Exchange for network authentication and e-mail

These systems were chosen because they deal with the business data at the root of the entire corporation and have the largest reach in terms of impact to the user community. The Employee Identity Synchronization system’s architecture was designed to be extensible to allow for inclusion of other existing and future systems.

Ceridian

Ceridian’s HR/Payroll is a human resources and payroll management system. It contains extensive amounts of information regarding the employee required by Human Resources to manage the employee’s life with the organization. It is the primary system for managing employee identity, since an employee must first be defined in the HR/Payroll system before they can exist in any other corporate system. From an architectural perspective, it is the "master system" for a number of the key Matrix Employee properties, such as the employee’s ID, first/last name, company, team, address, and status assignments.

Ceridian also contains information regarding employee benefits, dependant information, health and welfare data, and so on. Much of the data stored in Ceridian regarding Employee identity goes beyond the reach of the other systems, and therefore, from the perspective of the overall Matrix organization, is extraneous.

Because the system is hosted by Ceridian, and does not offer a Web service API, extracting the data from the system requires traditional-style file-based integration.

The data extract from Ceridian contains highly sensitive information such as employee names, addresses, and payroll data, so security was a critical architectural consideration.

Ceridian’s consulting services division constructed a CSV style file export of common properties derived from the data analysis exercise. The data file is encrypted and transmitted to a secure FTP server that is hosted at Matrix and specifically set up for the single purpose of receiving the extract file.

BST Enterprise

BST Enterprise is a Professional Services Automation (PSA) system that provides financial and project management tools for firms based in the Architecture and Engineering business. BST makes extensive use of employee identity information for tracking employee time and expenses, responsibilities and assignments on projects, and authorization to features within the BST system.

BST Enterprise therefore derives core identity information from Ceridian, but contains a large volume of information about the employee from a project perspective.

Ceridian views salary data from an annual context, and contains all sorts of information regarding reasons for a salary increase—whereas BST needs the salary information on an hourly basis in order to calculate and track project progress on a cost and budgetary basis. BST, therefore, is the owner of the properties associated with hourly project cost and charge rates. BST needs to know about salary increases in terms of the hourly cost rate, but does not need to know why these increases took place.

BST provides a .NET SDK (Software Development Kit) called the BST Freedom Framework™ that enables developers to programmatically interact with information in BST Enterprise in real-time. The SDK uses a document-centric model to define its business objects, and has strict validation rules about what is required to create a valid BST Employee. Therefore, key fields from BST are mapped into the Matrix Employee definition. In addition, to capture baseline information required by BST but not stored elsewhere, a "default" baseline employee was established as a root mapping point.

Active Directory/Exchange

Active Directory is used at Matrix as the primary authentication system, and Exchange Server is the e-mail system.

Because Matrix is very dynamic, employees will move about in the organization frequently. It is important that an employee’s status with the company is kept up to date in Active Directory and Exchange.

Since an employee’s team as defined by Ceridian corresponds to an Active Directory security group, and e-mail groups in Exchange, it was possible to create a set of rules that map the employee to the correct OU and E-Mail group automatically.

Datamart

The Datamart is a custom database that acts as the hub for all Employee Identity Synchronization workflow and mapping, including:

·         Staging area for Ceridian import data (the Datamart is used as a façade to represent Ceridian as a data source).

·         Mechanism through which the Enterprise Identity Synchronization system tracks new Matrix Employee objects through a key table. The key table also defines which systems are subscribing to the synchronization.

·         Central repository to store overall state of a Matrix Business object as derived through querying all subscribing business systems.

·         Holding of any information that is specific to Matrix and is not handled by other systems. Examples of employee information that is not provisioned by other systems are the Employee Chargeability and Hours targets. These targets are key performance indicators (KPI) that allow employees and business managers to track how employees are doing, with respect to goals that are set during performance reviews. These KPI are unique to Matrix, and, as a result, none of the commercial offerings has facilities for managing them.

Matrix Business Object XML Definition

Systems typically will take on the characteristics of the business domain that they are designed to serve. This has several architectural implications from the standpoint of identity management.

To leverage specialist knowledge associated with the business domain from a user experience perspective, systems will define identity objects using terminology associated with that domain. As a result, even identity information that is common across systems will be defined with different terminology and often on a different basis than is required by systems operating in a different domain.

Each system contains a great deal of information that is specific to the domain in which it resides, and has no purpose outside that domain. By going through each system in detail, we derived a definition of the union of identity that was outside each system domain. The resulting properties make up a business object that is called a "Matrix Employee." An XML-based standard was created to describe the Matrix Employee (see Figure 1).

Figure 1. Matrix employee business object XML

A map that defines the system owner of the properties (that is, the "master" property) and its name in the local system domain were established (see Figure 2). To limit the scope, the first iteration of the Employee Identity Synchronization system does not have a mechanism for reconciling differences between properties in different systems; therefore, a "master system" is established for each property, which tells the mapping engine which system to use as the source for a given property when synchronization takes place. A more sophisticated handling/reconciling system is planned for a later release of the system.

Figure 2. Matrix employee property ownership map

Implementing Identity Synchronization

BST Freedom Exchange (BSTFX) is both an extensible service framework and a service host for implementing and managing enterprise services based on SOA (service oriented architecture) best practices and principles (Figure 3).

Figure 3. BSTFX overview

BSTFX exposes a set of integration interfaces and APIs that give developers the freedom to define and implement completely customized services, service contracts, and workflows using WCF (Microsoft Windows Communication Foundation) and WF (Windows Workflow Foundation).

The Employee Identity Synchronization system is implemented as a set of BSTFX services, which are hosted by the BSTFX Runtime. The Runtime is responsible for registering and managing the services (Threading, Memory Management, Message Queuing, Activity Monitoring, and so on), while the Service Console provides administrators with the ability to manage and monitor registered services, including the ability to run and schedule jobs utilizing the custom service interfaces, inputs, and settings established by the developer (see Figure 4).

Figure 4. Settings in BSTFX console

The Enterprise Identity Synchronization process begins with Ceridian, which produces a nightly export of employee identity data to the Matrix SFTP site. Data is picked up by a scheduled SQL Server SSIS (SQL Server Integration Services) package, which is responsible for taking the data from the SFTP server, decrypting onto that server, and then downloading the file to a secure, temporary staging location on the SQL server, where it is imported into the Datamart (Figure 5). As an added security measure, the import process re-encrypts the salary information. E-mail alerts are sent out on success/fail of the import process.

Figure 5. SSIS integration package diagram

Identity synchronization involves taking the Matrix Employee XML object and asking each of the subscribing systems to fill the properties they own, validate them against the subscribing systems, and finally commit the final data back to each system. Doing this effectively builds a full picture of the current state of the Employee’s identity from all systems, and then commits the latest, composite version back to each subscribing system, thus bringing the systems into sync (Figure 6).

Figure 6. Enterprise Identity Synchronization structure

Interfaces

To isolate the specific requirements of each subscribing system from the overall synchronization pattern, the system is divided into two primary interfaces:

·         IMatrixDataSource—Describes the requirements for a valid synchronization data source and provides the basic methods for an object to fill its properties from the underlying data source, validate its properties against the Matrix Employee Business XML Object, and update any information that is mapped and synchronized from subscribing systems back into the local system domain.

·         IMatrixService—Describes the standard methods that must be implemented to define a valid Matrix Service that can be implemented to synchronize multiple data sources through workflow orchestration.

Workflow Services

The EmployeeQueue is exposed as a top-level workflow service that is managed by the BSTFX scheduler. The EmployeeQueue workflow process is responsible for iterating through the Ceridian employee identity list in the Datamart, determining which of the employees are new, generating a unique identifier for each new employee, and registering the related employee identifiers from each of the other systems.

Once the new employee identities are registered into the Datamart, the EmployeeQueue process calls the EmployeeSync service for each employee, passing the unique identifier assigned to the employee’s identity to the service.

Business Services

EmployeeSync loads the Matrix Employee Business Object XML definition, and then calls the SelectById() method on the Employee service, which passes the XML to the DataSource layer for each of the systems that is registered in the Employee Service.

Once the SelectById() workflow step is complete, the EmployeeSync workflow moves to the next step in the process and invokes the Update() method of the Employee service using the Matrix Employee XML derived from the participating systems.

The Update() method first runs a Validation() pass on all the Matrix Employee XML against all registered systems, which returns a MatrixBSTFXValidation object. If the validation passes cleanly for the employee identity record, the Update() method of the Employee Service calls the corresponding Update() method on each registered data source, passing in the Matrix Employee XML definition (Figure 7).

Figure 7. Employee Service Class diagram

Data services

Each data source provides a domain-specific implementation of Fill(), Validate(), and Update() methods of the IMatrixDatasource interface: (Figure 8)

Figure 8. DataSource classes

Figure 9. Synchronization interfaces

Figure 10. Synchronization pattern

·         ActiveDirectoryEmployee

·         CeridianEmployee

·         BSTEmployee

·         MatrixEmployee

The Fill() method maps the properties that are owned by the subscribing system into the Matrix Business Object XML. For example, the ActiveDirectoryEmployee data source will fill the E-Mail property, but will not fill the FirstName and LastName properties, which are owned by the CeridianEmployee data source.

Architecting the Enterprise Identify Synchronization system in this manner ensures that details of the mapping, domain-specific languages and queries are isolated from the top-level workflow and synchronization process. The top-level services and workflow services therefore only need to focus on the implementing requirements associated with handling the Matrix Business Object XML Each DataSource uses the Employee ID native to its data domain from the Matrix Employee XML. A blank value for a given ID means that the employee does not exist in the system in question—in which case, default values are returned.

If a system has knowledge regarding an employee’s identity, the DataSource layer maps the properties it owns from the local system domain into the Matrix Employee XML definition. This process builds the overall view of the employee’s identity from all systems registered in the Employee Service.

A successful pass through this process results in the synchronization of the Employee’s identity to all subscribing systems.

Conclusion

The Enterprise Identity Synchronization architecture provides Matrix with an automated framework for keeping identity up to date across multiple business systems and domains.

Defining an XML-based Matrix Business Object for data used by the synchronization architecture means that the services themselves become independent of the data they transport. This service/message pattern is the basis for creating a full Matrix Services Oriented Architecture. The use of a Matrix Business Object ensures that properties used across all systems are put into terminology that makes sense to Matrix Business, and is separate from specific system domains.

Because the BSTFX engine itself leverages Windows Communication Foundation, the actual service endpoints are separate from the communications protocol they use—so that it is possible to take advantage of having multiple entry points to each of the services (Figure 11). Separating the architecture into workflow and pure service layers that follow common patterns and contracts gives additional flexibility of being able to either call the workflow layer from an external client, or call the service layer directly.

Figure 11. Future systems

The enablement approach has also proven to be a powerful and effective blend of methodology, software architecture, and best-practice implementation, providing enormous business value and key benefits to Matrix, including:

·         The system was constructed from architecture to initial deployment in six weeks.

·         Leveraging BST’s expertise, while doing the actual production development onsite at Matrix, ensured that the system would directly meet Matrix requirements.

·         Because the engagement covered the entire software cycle from requirements through development, Matrix resources were able to participate in architectural decisions and more importantly build a full understanding of the system.

·         The base classes and contracts created for the Enterprise Identity Synchronization are generic enough to allow for extension of the framework not only to other systems, but to other classes of data, such as projects, clients, and vendors.

·         All materials, including source code, architectural drawings, and system-design documents, were turned over to Matrix—thus, giving Matrix full access to all assets that are derived from the engagement and the ability to extend the system.

The flexibility of the architecture is already being leveraged by several new Office InfoPath systems that query the Employee service layer and use the XML definition to extract identity data for lookup lists in the forms.

About the authors

Mike Morley is the Systems Architect at Matrix Solutions, a rapidly growing Environmental and Engineering Consulting firm based in Calgary, Alberta Canada. His responsibilities include designing and implementing corporate systems architecture, setting direction and vision for Matrix IT and IM based on Matrix business goals and system design, development, and deployment in support of and extending the business. Mike holds a Bachelor of Applied Sciences in Geological Engineering from the University of Waterloo. Mike has a diverse background in software development and design including developing three-dimensional blast design and drawing management applications for the mining industry, enterprise software development at Pandell Technology Corporation, and spatial/scientific software including 3D mine modeling systems at Golder Associates. Visit Mike’s blog at https://www.menome.com/.

Barry Lawrence is the Director of Systems Integration Management for BST Global and is responsible for driving the vision and development of the BST integration platform. Barry’s areas of expertise include Software Architecture, Quality Assurance, Project Management, and Software Development Methodology. Barry has over 15 of years of professional experience developing and leading technical teams in the implementation of enterprise-scale software systems, including seven years as an application development consultant for Software Architects (Sogeti, a subsidiary of Capgemini). Barry holds a Bachelor of Science in Marine Biology from the University of Tampa, and has certifications as a Microsoft Certified Solution Developer (MCSD) and Microsoft Certified Trainer (MCT).

This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal Web site.