Lead Hosts and Cluster Management (Windows Server AppFabric Caching)

Article
09/20/2010

A Windows Server AppFabric cache cluster is a dynamic group of servers working together to provide a single unified logical cache for your application's data. To make this happen, there is some overhead required to orchestrate the cluster operations between the cache hosts. The cluster management role is responsible for managing the cache hosts, and ultimately, the cache cluster.

Depending on how you deploy your distributed cache system, there are two options for what performs the cluster management role. If you store your cluster configuration settings in a SQL Server database, that SQL Server instance can also be used to perform the cluster management role.

If you choose to store your cluster configuration settings in a shared network folder, the cluster management role is always performed by special cache hosts, referred to as lead hosts. Lead hosts perform the same duties as other cache hosts not designated as lead hosts, but they have the additional responsibilities of working with other lead hosts to perform the cluster management role.

The following table shows how your choice at installation time relates to your options for cluster management. For more information about choosing which of these configuration options is right for you, see Cluster Configuration Storage Options (Windows Server AppFabric Caching).

Cluster configuration storage type	Cluster configuration storage location	Cluster management
XML file	shared network folder	lead hosts
SQL Server database	SQL Server	SQL Server (default) or lead hosts
Custom provider	custom store	custom store

Cluster Management Role Duties

There are two main configuration settings that determine how the cluster functions with respect to cluster management:

leadHostManagement: This cluster-level setting determines what will perform the cluster management role. When true, lead hosts perform the cluster management role. If you have chosen to store your cluster configuration settings in a shared network folder, true is the only valid value for this setting. False indicates that either SQL Server or a custom provider will perform the cluster management role. When using SQL Server or a custom provider to store cluster configuration settings, you can set this setting to true and let lead hosts perform the cluster management role.
leadHost: This cache host-level setting determines which cache hosts will be lead hosts when lead hosts perform the cluster management role. Even if SQL Server is going to perform the cluster management role, the installation program designates lead hosts, in case you later change the leadHostManagement setting.

For more information about changing these settings, see Set the Cluster Management Role and Lead Host Designations (Windows Server AppFabric Caching).

With these two properties, there are four possible cases to determine how a cache host will behave. These cases are described in the following table.

`leadHostManagement` cluster-level setting	`leadHost` cache host setting	Setting combination description	Effective cache host responsibilities
`false`	`false`	SQL Server or a custom provider performs the cluster management role. This is not a lead host.	Normal cache host operations only.
`false`	`true`	SQL Server performs the cluster management role. This is a lead host if you change the `leadHostManagement` setting to true.	Normal cache host operations only.
`true`	`false`	Lead hosts perform the cluster management role, but this is not a lead host.	Normal cache host operations only.
`true`	`true`	Lead hosts perform the cluster management role. This is a lead host.	Normal cache host operations and works with other lead hosts to manage the cluster.

When Lead Hosts Perform the Cluster Management Role

When the leadHostManagement and leadHost settings are true, the cache host is elevated to a level of increased responsibility in the cluster and designated as a lead host. In addition to the normal cache host's operations related to caching data, the lead host also works with other lead hosts to manage the cluster operations.

When a Lead Host Fails

For the cache cluster to remain available, a majority of lead hosts must remain available. This is more of a risk in small clusters than it is in large ones because it takes fewer server failures to cause the cluster to shut itself down.

Note

When lead hosts perform the cluster management role, if a majority of lead hosts fail, the entire cache cluster shuts down.

For example, consider the six-server cache cluster shown in the following diagram. In this example, lead hosts perform the cluster management role and two cache hosts have been designated to be lead hosts.

Cache cluster lead hosts

If any of the normal cache hosts in the cluster fail, the cluster could keep running. Data on the non-lead hosts would be lost (assuming high availability was not enabled), but the rest of the cluster could continue serving and storing data. In fact, the cluster could keep on functioning if it lost all four cache hosts not designated as lead hosts.

If just one of those lead hosts failed, the entire cache cluster would shut itself down because there would no longer be a majority of lead hosts running. To mitigate this issue, you do have the option of designating additional lead hosts.

Note

The Stop-CacheHost command will not stop a cache host Windows service if it is performing the cluster management role. The stop causes the entire cluster to shut down.

Designating Additional Lead Hosts

The AppFabric Configuration Wizard uses the Cluster Size drop-down list to help determine the appropriate number of lead hosts that should be in the cluster. If you would like to designate additional lead hosts after installation, you can do that also. However, it is important to consider that assigning too many lead hosts can also be a problem:

There must always be a majority of lead hosts available for the cache cluster to remain running. The more hosts designated as lead hosts, the fewer server failures the cluster will be able to sustain and remain operable.
In small clusters, where one or two lead host failures could cause the cluster to fail, we recommend that you designate more lead hosts.
In large clusters, five to seven lead hosts should be sufficient to ensure that a cluster in the range of 50 cache servers is responsive.

For more information about changing lead host designations, see Set the Cluster Management Role and Lead Host Designations (Windows Server AppFabric Caching).

When SQL Server Performs the Cluster Management Role

When the cluster's leadHostManagement setting is false, regardless of the leadHost setting, each cache host only performs its normal non-lead host responsibilities related to caching data. In this scenario, the instance of SQL Server that is used for storing cluster configuration settings is also used to perform the cluster management role.

When a Server Failure Occurs

For the cluster to remain available when SQL Server performs the cluster management role, one or more cache hosts must to be able to access the SQL Server database.

For example, consider the six-server cache cluster shown in the following diagram.

Cluster Management Role Set to SQL Server

In this example, SQL Server is performing the cluster management role, and all six cache hosts can dedicate their resources to data access for the cache clients.

If any one of the cache hosts in the cluster fails, the data on those servers is lost (assuming that high availability is not enabled), but the cluster keeps running. Data on the other cache hosts continues to be available to the cache clients. In fact, in this scenario, the cluster could keep functioning if it lost five of the six cache hosts.

If SQL Server fails, the entire cluster shuts down within a few minutes. To mitigate this issue, we highly recommend that you use Microsoft Windows Server 2008 Failover Clustering (https://go.microsoft.com/fwlink/?LinkId=130692) to host a "clustered" database resource for the cache cluster configuration storage location and cluster management role.