Designing Multitenant Applications on Windows Azure
Authors: Suren Machiraju and Ralph Squillace
Reviewers: Christian Martinez, James Podgorski, Valery Mizonov, and Michael Thomassy
This paper describes the approaches necessary to design multitenant applications (typically ISV applications that provide services for other organizations) for Windows Azure that are more efficient – that is, either less expensive to run or build, and/or more performant, robust, or scalable. The paper first describes the general principles of multitenant applications, the structure and behavior of the Windows Azure platform that is relevant to building and running multitenant applications, and then focuses on specific areas of multitenancy, especially as they apply to Windows Azure: application and data resources, as well as the provisioning and management of those resources. In each area of focus, this paper describes the security, cost, and scalability issues that you must balance to create a successful multitenant application on Windows Azure.
What is a Multitenant Application?
Wikipedia defines multitenancy in the following way.
|If you already know what we’re talking about in this article, go ahead and start reading the Windows Azure Multitenant Service Overview to get straight to the guidance. You can return to this section later if there’s something you didn’t quite understand.|
Multitenancy refers to a principle in software architecture where a single instance of the software runs on a server, serving multiple client organizations (tenants). Multitenancy is contrasted with a multi-instance architecture where separate software instances (or hardware systems) are set up for different client organizations. With a multitenant architecture, a software application is designed to virtually partition its data and configuration, and each client organization works with a customized virtual application instance. (Wikipedia.org)
What does this mean, simply put? Well, an example of this might be a restaurant reservation application. Restaurants themselves do not have the IT infrastructure to build a full-fledged internet reservation site that connects with the front desk in nearly real time to make or cancel reservations. (In most cases, restaurants hire other companies to build their static websites!) So you might create a single application that enables individual restaurants to sign up, create an account, customize the look of the web pages with their logos and colors, upload their menu, connect the web application with the computer at the front desk, handle reservation requests from other reservation applications, and so on.
In this example, each restaurant sees an application that they use, and that – once logged in – is only theirs. Their customers see only the web site of the restaurant, and for all they know, the restaurant built everything itself. (It’s this added value to the hungry customer that the restaurant is paying the ISV for!)This might look something like the following diagram, in which the Reservations Application supports multiple tenants – in this case, the Contoso and Fabrikiam Restaurants. In this diagram, Bob, Priyanka, Jasmine, and other customers are pondering whether to make a reservation or not, perhaps by looking at the menu.
It might look like the following diagram.
Architecturally speaking, it is possible to build such an application by hosting a brand new instance of it for each customer with special customizations for them; perhaps each customer gets its own virtual machine (VM). This is common enough, but it has many problems for the ISV, almost all of which appear once your application becomes reasonably popular (we’ll have a bit more to say on this later, but suffice it to say that this article does not focus on this approach, which is often referred to as a single-tenant, multi-instance design).
If you want your application to become popular among restaurants – and perhaps eventually among any company needing generic reservation support – you’ll want to build one application that shares resources among tenants (in this case, restaurants) such that your maintenance costs and resource costs are dramatically reduced, resulting in better performance and lower cost as you increase workloads. Building the application this way is what Wikipedia refers to as a multitenant, single-instance application.
The Application Instance
It is worth noting that “instance” in this sense refers to the entire logical application, and not a VM, or an .exe, a process, or anything like that. In fact, the Reservations application diagrammed above is composed of many different external and internal “services” – components that communicate through web services.
Such web applications are typically built to work with “web farming” – in which groups of servers handle multiple instances of important components of the application by storing any application state data in remote storage while executing, so that if any one instance of an application vanishes for any reason, another instance can either recover or complete the process. In short, modern, hugely scalable web applications are decomposed into process components and storage components that, along with communication and security components, together make up a single, outwardly facing “instance” of the overall application.
The Rise of Multitenant Services
Multitenancy on Windows Azure means decomposing applications into stateless services and storage and managing data flow by identity where necessary: State is only introduced in those services that must deal directly with the tenants’ identities. This follows the general service oriented application principle of Service Statelessness. For example, top-level services like web pages and public facing web services, or the security services themselves, of necessity must handle security claims and tokens directly. Other components and services that are part of the larger application, however, should strive to be reusable by any other portion of the application acting on behalf of any tenant.
Tenants and Their Customers Do Not Care About Multitenancy
This section is here to emphasize one thing: Tenant companies and their customers do not care how your application is built. They care ONLY about how the application works: its robustness, performance, feature set, and cost to them. If they care about other things, they’re just nosy geeks. (We shall concede there are indeed cases in which you may have to demonstrate through an audit of some kind that you really ARE building and running things correctly for security and especially legal or regulatory reasons. But these are only the exceptions that prove the rule; those audits are for compliance, not architecture per se.)
Windows Azure Multitenant Service Overview
The key problem designing a multitenant system is deciding upon the right balance between providing tenant isolation (on the one hand) and the cost of providing such dedicated resources. Said a different way, you must figure out just how much of the resources can be shared amongst your tenants so that your solution works properly but is as cost efficient as possible. Typically, the more a resource can be shared amongst many tenants, the more cost efficient that solution is – so long as it still satisfies the performance and security requirements.
|If at any time you’re not clear about what multitenancy is or why it’s important for Windows Azure applications, read What is a Multitenant Application? to get some context; then you can return here.|
Within that larger problem, the major considerations center around:
Security - Isolating stored data, authentication & authorization mechanisms
Scaling - Auto-scaling platform compute, scaling platform storage
Management and monitoring
Provisioning tenant resources
Metering and billing
Observe that devising a multitenant architecture is an involved process, with many moving parts and this article does not attempt to address every detail, but rather put together a perspective on the major considerations with as many commonly helpful tips and pointers to the details as possible. With those points in mind, let’s start to explore the architecture from a high level.
High Level Architecture
One approach to supporting multiple distinct customers is to reject the multitenant route completely and allocate resources for each customer on demand. When you use this approach – called single tenant -- you are automating the provisioning of dedicated resources for customers. If you imagine a spectrum of distributed application architectures, this approach represents completely tenant-dedicated resources on one side, with completely shared resources on the other. This paper attempts to describe how to design distributed applications that are closer to the completely shared portion of this design spectrum: distributed applications with shared (or multitenant) application and data resources.
When you build a multitenant application, you need to weigh the costs of resources against the isolation you want to provide each customer. It’s easy to think that you will need to consider which application and data resources are shared between customers and which are allocated exclusively to customers, but it's important to realize that being multitenant is not an all-or-nothing proposition when it comes to resource sharing. In a service-oriented application, you are able to – and most likely will -- mix and match which resources are shared between customers and which resources are exclusive to individual customers depending on your requirements. It’s worth emphasizing this again: You can decide which application services can be shared, and which cannot be – and to what extent they are shared. This gives you a tremendous amount of design flexibility. Use it.
Which Resources Can Be Shared?
The figure below illustrates the Windows Azure resources under consideration, and we’ll explore some of the decisions behind each of these as motivation. One helpful way to look at all the resources in the diagram (and multitenant architectures in general) is to take each component and define where it sits within the application or data tier.
In this diagram, Windows Azure compute and the Access Control Service are clearly application layer resources, as are relay endpoints, which serve to connect computing instances. Tables, Blobs, and SQL Database are clearly data layer resources. Caching, however, along with queues, often serves as temporary storage that enables certain application designs and communication patterns, and as a result can be seen as both storage or as application layer resources, depending upon their usage.
Furthermore, exactly which approach you can take in any one case depends upon how many features you intend to expose to your tenant. For example, if you want to allow your tenant to offer a tenant-specific identity to its clients, you may want to have a general – and multitenant-safe – username/password security token service (STS). But you might also enable your tenant to implement one, and federate with Access Control Service instead. Another example is whether you intend to allow your tenant’s customers to upload and execute code in the tenant’s web or worker role instances. Doing so may well require a particular multitenant solution that mixes and matches multitenant and single-tenant role instances.
With these ideas in mind, let’s consider how to successfully share the resource. Although we’ll discuss in this and other topics the general approaches you can take to supporting multiple tenants in one resource from the point of view of an application, we’ll also discuss how to provision and manage those resources, and how to use Azure to do so. Provisioning and managing resources that are shared does not pose great problems, but in some cases it’s worth a closer look.
Designing a multitenant application means making sure that to the extent possible compute, data, and other application components handle multiple tenant identities and potentially each tenant’s customer identities as well. This section discusses the issues involved in various approaches using Windows Azure and suggests the best approaches to handling them in your application design.
Let’s begin by looking at the application resources provided by Windows Azure web and worker roles, and how they should be used in a multitenant solution to host web services, web sites, and other general processing.
The clearest use of Windows Azure web and worker roles in a multitenant application is to provide additional (or to reduce!) compute resources on demand in the service of the tenants, with the notion that a single role instance serves multiple tenants.
Segment Web site Tenants by Host Headers
In the most coarse-grained solution, the approach is to configure each web role instance to support multiple web sites and web applications, perhaps one for each tenant. This can be accomplished by modifying the Sites element in the Service Definition. Each website can either be bound to the same endpoint and use the HTTP/1.1 Host header to isolate them or they can be bound to their own endpoints. Note, however, that binding each web site to its own endpoint doesn’t scale per tenant since Windows Azure limits you to 5 input endpoints per role instance).
If you use HTTP/1.1 Host headers to support multiple tenants, the tenants (and their customers) simply visit a different a URL (such as www.fabrikam.com for one and www.contoso.com for the other tenant) and get the website for the desired tenant, even though they are in fact to talking to a single endpoint within a single web role instance. This approach allows you to isolate different web site files by tenant with minimal effort. As an additional benefit, this approach enables you to isolate tenants by application pool because by default each web site gets its own application pool. Note, however, that because the URLs that provide the HTTP/1.1 Host headers are defined in the domain name system (DNS), you will need to configure the DNS with your domain provider and point it to your *.cloudapp.net address; remember that it may take 24 hours for your new tenant URL to be fully discoverable on the internet. For a great example of how one instance can host multiple websites differentiated by host headers, check out the Windows Azure Accelerator for Web Roles.
One cautionary note to using the HTTP/1.1 Host headers approach is that Windows Azure does not enforce a security boundary between these websites; this means that SSL certificates used by one website are useable from the other websites. To complicate matters, IIS 7 does not support having multiple websites with their own certificate if only their host headers are different.
|It is possible to have multiple SSL Web sites that use unique server certificates if each Web site uses a separate IP address or port for its HTTPS binding. As in IIS 6.0, IIS 7.0 does not support having multiple Web sites with their own server certificates if the HTTPS bindings for those Web sites are on the same IP address/port and differentiate only by using host headers. This is because the host header information is not available when the SSL negotiation occurs. Because of this, the only way to have multiple Web sites on the same IP address that use host headers is to configure all of those Web sites to use the same SSL certificate with a wildcard CN. For more information, see http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/596b9108-b1a7-494d-885d-f8941b07554c.mspx?mfr=true.|
Therefore, if you are using SSL certificates, you may need to take a different approach. One is to use a single certificate with a wildcard CN (of the form *.yourdomain.com) or a Unified Communications Certificate (where you list all the subdomains explicitly). These types of certificates are available from certificate authorities like GoDaddy and Thawte and are straightforward to create. Each tenant would then access resources over SSL using URL’s like https://fabrikam.yourdomain.com or https://contoso.yourdomain.com.
The alternative is to have each website live on a dedicated web role and only the one tenant certificate. As this is not a multitenant approach, it does not scale as well and therefore absorbs more compute resources as well as management overhead. It nevertheless must sometimes be done.
Segment Website Tenants by Query Parameters
Another approach is to use a single website to serve resources for multiple tenants. If your website is built using ASP.NET MVC, you could create a separate MVC area for each tenant. Alternatively, you share the default area between them and instead have the controllers render differently based on query parameters (such as a tenant id). If you want to provide process isolation by tenant, then you need to define virtual applications within the website. Effectively, each tenant gets a virtual application underneath the website. This is also accomplished by editing the Sites element within the Service Definition, but instead of adding a new Site for each tenant, within your main Site element you add one VirtualApplication element for each tenant.
Web Services in Worker Roles
For fully multitenant solutions, worker roles hosting web services behave the same way as their relatives hosted in a web role—all tenants access the same service endpoints.
Worker Roles are also used to provide background processing. In the fully multitenant scenario, the approach you take is to have a worker role whose RoleEntryPoint.Run method loop processes work items agnostic of which tenant created it -- hence you are sharing the compute resources across all tenants.
Multitenant applications must handle how the data they store and use is isolated and yet performs well. This section discusses data storage technologies in Windows Azure, including SQL Databases; Windows Azure blobs, tables, and queues; Service Bus queues and topics, and the Caching service in a multitenant application.
SQL Database storage is clearly an important location in acting as the storage for an application, but there use in a multitenant architecture extends to acting as storage for provisioning new tenants and for storing management data.
Using SQL Database for Application Resources
In this scenario application data for multiple tenants are stored within a single logical database. While there are obvious cost benefits to delivering a multitenant database (realized by splitting the cost across all tenants), these come with the tradeoff of increased complexity in dealing with various requirements specific to this scenario:
Adequately isolating data and preventing one tenant from inadvertently or maliciously accessing another tenant’s data.
Maintaining reasonable query performance for all tenants, in addition to limiting the effect of a single, especially resource-hungry tenant.
Handling the rapidly increasing size of a shared database, which may exceed the maximum 50 GB database size that SQL Database currently supports.
Conceptually, there is a performance benefit gained when distributing tenant data and query processing across multiple databases—the question becomes, what architectures allow you to leverage this for your solution? Before we answer that question, it’s is helpful to look at the architectures on a spectrum. On one end of this spectrum is the traditional scale-up, in which you simply throw bigger hardware at the problem. As with the single-tenant architectural approach, we won’t focus on scaling-up except where it helps understanding, because it is too limiting for multitenant solutions. We focus on the various scale-out approaches using sharding techniques:
Compressed Shard and
Hybrid Linear/Expanded Shard architectures.
All of the preceding four useful sharding techniques add databases to the architecture to support scale-out, and for each there is some logical separation between the data stored (often referred to as a data domain or data federation) that defines the shard. These data domains can be defined using certain key values (such as a customer id) or an attribute (such as the year). For the sake of simplicity in this article, and in the context of multitenant architectures, we take the perspective that the data domain is the tenant.
Returning to the spectrum diagram, on one end of the spectrum we have the traditional scale-Up model. In this approach all tenants share the same SQL Database. In other words we have multiple tenants per database. In the middle of the spectrum diagram, above, we have the linear shard architecture, in which each tenant gets its own SQL Database. On the far end of the spectrum, each tenant may get multiple SQL Databases, which is referred to as the expanded shard approach.
The other two approaches address in-between scenarios. In a compressed shard architecture there are fewer SQL Databases than tenants, but more than one SQL Database. Similarly, scenarios where there are multiple data domains with which sharding decisions are made (such as a region and tenant), then we have Hybrid Linear/Expanded Shard architecture where each region gets a particular expanded shard each shard having multiple databases, and a particular tenant’s data is spread across multiple databases in that expanded shard. In other words, the architecture scales out linearly by region, but in an expanded fashion by tenant.These sharding approaches necessitate a need for a layer of abstraction, like the Enzo SQL Shard Library, that knows how to navigate the sharding approach implemented, and in particular assists with fanning out queries to all shards and guiding data modifications operations to the appropriate shard.Windows Azure SQL Database Federations coming out the second half of 2011 promises to simplify the implementation of sharding architectures implemented on SQL Database, but let’s consider what steps you can take to implement these architectures today.
Securing Multitenant Data in SQL Database
We begin by looking at how to secure your tenants data. In its core form, managing security at the database level for SQL Database is almost identical to that of an on-premise SQL Server, amounting to creating security principals and using them to control access to specific schema objects (schemas, tables, views, stored procedures, and so on). Then you create the appropriate per-tenant SQL Server logins, Database Users, and Database Roles, and apply permissions to schema objects using these security principals. One approach is to define one schema per tenant – security applied to schemas restrict access to just that tenant.
Another approach is to store all tenant data within the same set of tables, separating them within the tables by some key (such as a tenant id column), thereby creating a logical separation. In this approach logic in the business tier is responsible for correctly filtering data accesses by tenant (albeit SQL Database Federations can also perform this task). The crux of this approach is enforcing that all requests for data must go through this logic in order to limit access to the correct tenants; OData (using WCF Data Services that use Query and Change Interceptors) is a great technology with which to implement this approach. There are even some frameworks, like Enzo (discussed later) that relieve you from having to implement this logic.
With this in mind, let’s apply this to the scale-out architectures previously described. Security in the Linear Shard architecture can be pretty simple: each tenant gets its own database, and as such gets its own dedicated logins and users specifically for that shard. In all of the other sharding architectures, security needs to be more granular and relies on a combination of permissions defined at the schema object level and enforcement at the business tier.
Next, let’s turn our focus to performance. SQL Database maintains three replicas of any SQL Database. While data for each database is replicated between one primary and two secondary replicas, all reads and writes take place on the primary replica- the secondary replicas only help out with query processing in a fail over scenario. The Windows Azure SQL Database fabric determines the location of the primary and two secondary replicas. If a machine hosting your primary replica is under heavy load (due to your own tenants or other tenants), the SQL Database fabric may switch the primary for a secondary on another machine that has a lighter work load. This switch happens quickly, but does result in the disconnection of active sessions—a situation which your multitenant application must be prepared to handle.
With this in mind, if per-tenant performance is a top priority, then you should consider a sharding architecture that allocates one or more databases per tenant (such as the Linear, Expanded or Hybrid architectures). However, if you have a lot of tenants for which you allocate a database this way, you run into a few obstacles. Obviously, each database you create has a cost, but the obstacle to be aware of is the limit of 149 databases per SQL Database Server, which can be raised if you call Cloud Support Services (CSS). For multitenant scenarios with lots of databases, your solution also needs to be able to allocate SQL Database Servers as this limit is approached.
Using Windows Azure Tables for Application Resources
Windows Azure Tables are secured using a storage account name and key, which can only be allocated globally for all storage resources. This means that to provide isolated storage of per-tenant data the service provider will have to create a storage account for each tenant. This can be performed through the Windows Azure Service Management API. There is no additional cost to allocating new storage accounts and there is a performance benefit to be gained by doing so because different resources can be applied to different accounts. That said, there are quotas governing the number of storage accounts a subscription can have, which can be adjusted by contacting Cloud Services Support (CSS).
If your application needs to store multiple tenants’ worth of data within a single account, you have to create your own abstraction to filter the data returned to a given tenant. One approach is to create one table per tenant, and authorize access at a per table grain. This approach is beneficial in that you get to leverage the full partition key and row key to quickly access the data, with a fairly simple logic for routing queries to the correct table. The downside is the extra complexity in that you have to route queries to the appropriate table based on the tenant. Another approach is to partition horizontally by tenant within the table, where your data access layer only returns data having a certain partition key to the client. This eliminates the need to juggle routing queries to the appropriate table, but limits you to a single partition key value, which in turn can affect your performance on very large per-tenant datasets.
Using Windows Azure Blobs for Application Resources
When it comes to the storage needs of the application, Windows Azure Blobs offer a great deal for multitenant applications. You should create public blob containers for read-only access to shared data like website images, graphics and tenant logos. Private blob containers (only accessible by the tenants or the tenants and the system) would be used for application data like documents or per-tenant configuration files. Outside of anonymous access, they key approach is to be sure to specify a container-level access policy for containers or blobs secured using Shared Access Signatures for various actions such as blob read/write, block list, properties, or metadata, deleting a blob, leasing a blob and enumerating blobs within a container. By specifying a container level access policy, you can the ability to adjust permissions without having to issue new URL’s for the resources protected with shared access signatures.
Windows Azure Queues for Application Resources
Windows Azure queues are commonly used to drive processing on behalf of tenants, but may also be used to distribute work required for provisioning or management.
Here, the consideration is whether a given queue manages items belonging to multiple tenants, or if each queue is dedicated to a single tenant. The Windows Azure Storage model works at the storage account level only, therefore if queues need to be isolated by tenant, they need to be created under separate storage accounts just as for tables. Moreover, in the case of a single queue, it’s not possible to define permissions restricting use to only put or only receive messages—once a tenant has access with the storage account the tenant can do everything allowed by the Azure Queue API, even delete the shared queue! In short, the queues should not be exposed to the tenant, but rather managed by the service automatically.
Service Bus Queues for Application Resources
For tenant specific application functionality that pushes work to a shared a service, you can use a single queue where each tenant sender only has permissions (as derived from claims issued from ACS) to push to that queue, while only the receivers from the service have permission to pull from the queue the data coming from multiple tenants. The reverse direction of this is possible with AF Queues, you can have system services push messages destined for specific tenant receivers across a single queue. You can also multicast these messages by using the Subscription functionality of AF Queues. Finally, you can use Session Messages to drive customer specific work to customer-specific receivers, albeit only pushing through a single, shared queue.
While these approaches minimize the number of queues you have to manage (which to a certain extent reduces your overall system complexity), it can limit the throughput of enqueue or dequeue operations because the load is not partitioned amongst more resources (as would be the case when using multiple queues). Also, in the case of multiple tenant senders, you must be careful not to be at the mercy of a single tenant who floods the queue and effectively starves other tenants of downstream processing. With AF Queues you can defer messages, so it is also possible to detect such a flood coming from a single tenant, defer those messages temporarily to process messages from other tenants and later return to processing them.
Cache Service for Application Resources
Within a multitenant application, the Cache is traditionally used for frequently accessed tenant-specific application data. As it does not provide the ability to set distinct permissions within a single cache, the pattern is to provision a distinct cache per tenant. This implies provisioning a new AppFabric namespace for the tenant and creating the cache within that. At present, there is no management API to automate Cache creation and this still needs to be done by human operator using the Windows Azure Portal.
Connection and Security Services
Windows Azure multitenant applications use other “middleware” services for connectivity and security: Service Bus relay services and the Windows Azure Access Control Service (ACS).
Using ACS for Application Resources
Windows Azure Access Control Service
is a cloud-based service that provides an easy way of authenticating and authorizing users to gain access to your web applications and services while allowing the features of authentication and authorization to be factored out of your code. Instead of implementing an authentication system with user accounts that are specific to your application, you can let ACS orchestrate the authentication and much of the authorization of your users. ACS integrates with standards-based identity providers, including enterprise directories such as Active Directory, and web identities such as Windows Live ID, Google, Yahoo!, and Facebook. (If you don’t believe us, read about it here.)
Whether you are building a multitenant system or not, ACS is the way to secure access to the application itself. Using ACS enables your application to be built using one authentication API (the ACS API) that can process security tokens from any identity provider. This means you don’t write separate code each for Facebook, Google, Yahoo, Windows Live, a different third-party identity provider, or Active Directory Federation Server (ADFS). Instead, you only write code for ACS, and let ACS and the identity providers do the heavy lifting for you.
Provisioning Identities in ACS
The details of configuring ACS or implementing your own STS are beyond the scope of this document, but you can get started thinking about such things in Authorization in Claims-Aware Web Applications and Services. To get the fundamentals of the approach used to outsource security from your application via federated security, see this great guide. To get a start on building your own, you might want to get the background on how to build one using Windows Identity Foundation (WIF) from here, but you will likely benefit by starting with a fully fleshed out STS like Thinktecture’s Starter STS, and customizing it to suite your needs.
Within a multitenant application that uses ACS, identities are usually provisioned with the following high-level steps:
Creating certificates (for example, per tenant)
ACS namespace provisioning (if isolating by namespace), including the configuration of tenant’s application that is to be secured -- called a relying party (RP) -- and claims transformation rules. This is done using ACS management APIs or the ACS Portal.
Creating root administrator account for tenant to login with (using API of an identity providing security token service).
Using Service Bus Relay for Application Resources
The services that are exposed as endpoints may belong to the tenant (for example, hosted outside of the system, such as on-premise), or they may be services provisioned specifically for the tenant (because sensitive, tenant-specific data travels across them). In either scenario, handling multiple tenants is really not the issue; enforcing tenant-specific usage is. Access to these endpoints is secured using Access Control Service (ACS), where clients must present an Issuer Name and Key, a SAML token, or a Simple Web Token. This can be configured programmatically using the Service Bus and ACS management APIs.
|Remember that Service Bus queues and topics and subscriptions are discussed as storage, though they often serve to handle application data flow in a particular way.|
This section discusses the provisioning of resources in a way that supports multitenant applications. As with supporting more than one tenant in application design, designing multitenant provisioning also has several decision points, depending upon the features you intend to enable and what Windows Azure services your application uses.
As with the design of multitenant applications, we’ll discuss how compute, storage, and connectivity and security services are used to provision multitenant applications.
Provisioning and Managing Resources Using Azure Roles
A dedicated worker role in multitenant solution is typically used to provision and de-provision per tenant resources (such as when a new tenant signs-up or cancels), collecting metrics for metering use, and managing scale by following a certain schedule or in response to the crossing of thresholds of key performance indicators. This same role may also be used to push out updates and upgrades to the solution.
A web role is typically dedicated to enable the service provider to monitor and manage system resources, view logs, performance counters, and provision manually, and so on.
Using ACS for Provisioning Resources
When provisioning tenant resources protected by ACS, you will need to use the ACS management APIs or portal to create the initial admin" account for newly provisioned tenants.
Using Cache Service for Provisioning Resources
When provisioning tenant resources protected by ACS, you will need to use the ACS management APIs or portal to create the initial admin" account for newly provisioned tenants.
Considerations for Provisioning Storage
One consideration that applies to Windows Azure Tables, blobs, and SQL Database in a multitenant solution is geo-distribution. Specifically it is identifying data that must be unique system-wide (across all data centers used in the solution) and balancing that against maintaining a performant user experience. One option is to build a custom replication strategy to bring such shared data near to end users (which naturally has to ensure that new data is unique, perhaps by always inserting to the master data source). The other option is to partition the data, such that the amount and frequency of access of global data is minimized.
Another consideration for Azure Storage in general is the hard limits imposed by Azure, which albeit large are not infinite. When planning your multitenant solution, you should take these scalability limits into account.
Using SQL Database for Provisioning Resources
In some multitenant scenarios with large amount of data involved, it is best to provision new SQL Databases by copying from an existing SQL Database reference instance. The enhanced provisioning speed provided by this must be weighed against the cost of maintaining an extra SQL Database on top of those required by tenants and the system itself.
Provisioning SQL Database Resources
The options for provisioning new SQL Database resources for a tenant include:
Use DDL in scripts or embedded as resources within assemblies
Create SQL Server 2008 R2 DAC Packages and deploy them using the API's. You can also deploy from a DAC package in Windows Azure blob storage to SQL Database, as shown in this example.
Copy from a master reference database
Use database Import and Export to provision new databases from a file.
Provisioning Windows Azure BLOB Storage
The approach for provisioning BLOB storage is to first create the container(s), then to each apply the policy and create and apply Shared Access Keys to protected containers and blobs.
Using Windows Azure Blobs for Provisioning Resources
When it comes to provisioning compute or pre-initialized storage resources for new tenants, Azure blob storage should be secured using the container level access policy (as described above) to protect the CS Packages, VHD images and other resources.
Finally, the design of a multitenant application must tackle the extremely important task of managing the application, tenants and their services, all the data resources, and any connectivity or security issues they entail. This section discusses common uses of compute, data, and other services to support multitenant applications while running Windows Azure.
Using Windows Azure Roles for Management Resources
The service provider will need a way to monitor and manage the system resources. A web role is typically dedicated to provide the service provider with tools to manage resources, view logs, performance counters, and provision manually, etc.
Using ACS for Management Resources
Most multitenant systems will require a namespace created within ACS that is used to secure system resources, as well as the creation and management of individual namespaces per tenant (such as for using the AF Cache). This is also accomplished using the ACS management namespace.
Using Cache Service for Management Resources
If the service provider exposes certain KPI’s or computed statistics to all tenants, it may decide to cache these often requested values. The tenants themselves do not get direct access to the cache, and must go through some intermediary (such as a WCF service) that retains the actual authorization key and URL for access to the Cache.
Using SQL Database for Management Resources
Examples of these are single, system wide and datacenter agnostic membership/roles database for non-federating tenants or those relying on a custom IP STS configured for use with ACS. For multitenant systems concerned with multiple geographic distributions, the challenge of a centralized system for management data surfaces. To solve these problems you can take the approach previously described for application resources, either by defining your geographical regions as shards in Hybrid Linear/Expanded Shard architecture or a more simple Linear Shard architecture. In both cases, leveraging middle-tier logic to fan out and aggregate results for your monitoring and management queries.
Using Windows Azure Tables for Management Resources
The Windows Azure Diagnostics (WAD) infrastructure by default logs to Windows Azure Tables. When relying on these WAD tables (Trace, Event Logs and Performance Counters) you need to consider just how sensitive the logged data may, who has access to them and ultimately choose if they are isolated (aka provisioned) between customers or shared system-wide. For example, it’s unlikely that you would provide all tenants direct access to the diagnostic log tables which aggregates traces from all instances across the solution and might expose one tenant’s data to another tenant.
Using Windows Azure Blobs for Management Resources
The canonical management resources stored in blob storage are IIS Logs and crash dumps. IIS Logs are transferred from role instances to blob storage. If your system monitoring relies on the IIS Logs, you will want to ensure that only the system has access to these logs via a container level access policy. If your tenants require some of this data, you will want to perform some post processing on the data (perhaps to ensure that only that tenant’s data is included) and then push the results out to tenants via a tenant specific container. Crash dumps, on the other hand are something only the service provider system should have access to as they assist in troubleshooting the infrastructure and will very likely contain data that spans tenants.
Within the context of multitenant applications, metering is motivated by a desire to bill charges to tenants that are influenced by usage as well as to collect system-wide KPI's to inform capacity planning and on-going architecture decisions. What is metered? Typically it boils down to these:
Raw resource consumption: Azure and AppFabric resource use (e.g., compute hours, data storage size, number of messages)
Specific use of applications features (for example, premium operations charged per use)
Use by tenants own users
The latter two tend to be driven by application specific requirements, so we will focus on the raw resource consumption that is common to all Azure based service providers. From a raw resource perspective, for most SaaS applications, compute hours is by far the most significant, followed by storage size and to a lesser degree, data transfer out from the data centers (egress) -- particularly as data transfers into Azure datacenters is now free.
So how can you get this data for metering on raw compute or storage? Let’s explore some examples.
Unfortunately there is no API at present to query for this by the service provider's subscription. The best approach is to approximate usage in line with how Windows Azure compute time is billed. For example, for each instance allocated, compute the hours of instance uptime, rounding up to the nearest hour multiplied by the hourly cost for the instance size.
Again, there is no public billing or management API for Windows Azure Storage (Blobs, Tables, and to a lesser extent Queues) or Cache and Service Bus Queues) that provides exact usage, but one can extrapolate by knowing the size of the entities being stored and tracking the number of entities stored by the tenant on average by the tenant over the billing period. SQL Database size is the exception; You can determine the database size that for which you will be billed that day by querying sys.database_usage within the master database and aggregating the result over the month to get at the actual cost.
Scaling Compute for Multitenant Solutions
While specific requirements vary, as a general theme when "auto-scaling" is considered the approach amounts to increasing or decreasing the instance count according to some heuristic. This heuristic may depend on some key performance indicator (KPI) derived from sources such as performance logs, IIS logs or even queue depth. Alternately, it may simply be implemented in response to a schedule, such as incrementing for a month end burst typical in accounting applications, and decrementing the instance count at other times.
The key factor to consider here is that scaling the instance count up or down is not instantaneous. Your algorithms should incorporate this in two ways. When scaling up, it’s best if you can anticipate the demand-- it doesn't have to be a long range forecast (but on the order of 20 minutes) to allow for your instances to become available. When scaling down, recall that you pay for partial hour used as a complete hour, so it makes economic sense to keep those unneeded instances around for that full hour.
Conclusion and Resources
Frameworks and Samples
Having read through the above, you probably agree that building a multitenant solution is a big investment. From this perspective, starting from a sample or framework is a great idea. Here are some great starting points.
Microsoft Enterprise Library 5.0 Integration Pack for Windows Azure
Microsoft Enterprise Library is a collection of reusable application blocks that address common cross-cutting concerns in enterprise software development. The Microsoft Enterprise Library Integration Pack for Windows Azure is an extension to Microsoft Enterprise Library 5.0 that can be used with the Windows Azure technology platform. It includes the Autoscaling Application Block, Transient Fault Handling Application Block, blob configuration source, protected configuration provider, and learning materials. This application block is a great place to start.
The CloudNinja Sample available on CodePlex demonstrates the following aspects of multi-tenancy as discussed in this article.
Multitenant Web Roles
Dedicated Management/Provisioning Worker Roles
Linear Sharding in SQL Database
Dedicated Windows Azure Tables for Management
Dedicated Azure Queues for Provisioning
Dedicated Public/Private Blob Storage
Multitenant AppFabric Cache
Time & KPI Based Auto-Scaling
Federated Security with a Custom STS and ACS
The Fabrikam Shipping Sample sample available on Codeplex demonstrates many of the aspects of a multitenant SaaS offering.
Dedicated & Multitenant Web Roles
Dedicated Management/Provisioning Worker Roles
Linear Sharding in SQL Database
Dedicated Public/Private Blob Storage
Federated Security with a Custom STS, ADFS, Facebook, Google, Live ID and ACS
Enzo SQL Shard Library
The Enzo SQL Shard Library available on CodePlex demonstrates and assists with the numerous SQL Database sharding techniques discussed in this article.
Lokad Cloud is a feature rich framework that provides lots of the functionality as described in this article and a lot more.
Strongly typed Azure Blob, Table and Queue storage (they call this an Object to Cloud mapper).
Azure Auto Scaling
If you’re looking for a simple, straightforward example from which to build your own auto-scaling, the Azure Auto Scaling sample available on CodePlex is worth a look.Of course, there are also some commercial offerings available to help with some of the more difficult aspects. In particular, Paraleap AzureWatch can help you auto-scale your application.
Links and References
Developing Applications for the Cloud on the Microsoft Windows Azure™ Platform
Architecture Strategies for Catching the Long Tail
Multi-tenancy in the cloud: Why it matters
The Force.com Multitenant Architecture: Understanding the Design of Salesforce.com’s Internet Application Development Platform
Patterns For High Availability, Scalability, And Computing Power With Windows Azure
GeekSpeak: Autoscaling Azure
Performance-Based Scaling in Windows Azure
Accounts and Billing in Windows Azure SQL Database
Security Resources for Windows Azure
How to Configure a Web Role for Multiple Web Sites
How to Configure a Web Role for Multiple Web Sites
Federations: Building Scalable, Elastic, and Multi-tenant Database Solutions with Windows Azure SQL Database
Managing Databases and Logins in Windows Azure SQL Database
Inside Windows Azure SQL Database