Throttling

  • Article

When physical memory becomes low on a Microsoft AppFabric 1.1 for Windows Server cache host, the cache host can enter a state called throttling. The cache cluster will not write data to any cache that resides on a throttled cache host until the available physical memory increases to resolve the throttled state.

Diagnosing Throttling

The most obvious symptom of throttling will come from applications. Attempts to write to the cache will generate DataCacheException errors. For more information about the exception, see RetryLater (Throttled). To confirm that throttling has become an issue on the cache cluster, you can use one or more of the following tests.

  • Use the Get-CacheClusterHealth Windows PowerShell Command. Look to see if any of the cache hosts have caches with percentages in the Throttled category.

  • Enable the Operational log in the Event Viewer on each cache host, and look for Event 116 which states "Service in throttled state".

  • Use the Performance Monitor to track the Memory | Available MBytes on each cache host. When this gets under 15% of total physical memory, the cache host enters the throttled state. Throttling also occurs when the cache host memory comes within 4% of the CacheSize setting.

Note that a server in the throttled state will also show increased eviction runs. For more information on eviction problems, see Eviction.

Note

For more information about the tools described above, see Health Monitoring Tools (AppFabric 1.1 Caching).

Resolving Throttling

Before deciding on a resolution to throttling, it is important to better understand why it occurred. The table below provides a possible cause and the recommended solution.

Throttling Cause Description and Resolution

Other processes using too much memory.

It is possible that other processes on the cache host are using large amounts of memory. This could result in low-memory conditions depending on the amount of memory reserved for the Caching Service. You can detect this by using the Process | Private Bytes counter in Performance Monitor to view the memory used for each process. If the Caching Service, DistributedCacheService.exe, is not the primary memory consumer, look for other processes that are consuming a high percentage of memory. One solution is to move the problem processes to other servers that are not cache hosts. You can also add more physical memory to the machine.

Note

Although you can use Set-CacheHostConfig to set the cache size for a cache host, this limit only determines when to trigger eviction runs. It does not guarantee that the memory for the Caching Service will stay at that level. For more information, see Eviction.

Caching Service using too much memory.

This can be identified by using Task Manager or Performance Monitor to view the memory used by the Caching Service, DistributedCacheService.exe. It is possible that one or more of the caches have disabled expiration and/or eviction. This can contribute to the low-memory condition. You find out the caches in the cluster by using the Windows PowerShell command Get-Cache -MaxRegions 0. You can look at the settings per cache with the Get-CacheConfig command. For more information about expiration and eviction, see Expiration and Eviction. It is also possible that the demands on the cache cluster are exceeding the capacity. You can add more physical memory to the cache hosts, or you can add additional cache hosts to the cluster.

Uncollected .NET memory.

Although .NET garbage collection occurs automatically, there may be times between collections where uncollected memory is contributing to the low-memory conditions of the cache host. In this scenario, use the Windows PowerShell command Invoke-CacheGC on the throttled cache host to force a full garbage collection cycle. If the low-memory conditions persist, then uncollected .NET memory was not a contributing factor.

Custom regions.

Application can create custom regions with the CreateRegion method. These regions always exist as a unit on a specific cache host. If the application puts large amounts of data in a single region, it is possible that one cache host will become throttled even though there is available memory on other cache hosts. One solution is to add more memory to each cache host in the cluster, because you cannot know for sure which cache host will be chosen for the custom region. The other solution is to redesign the application to store less data in the region or to create and use multiple regions.

Tag hash tables

Several AppFabric Caching methods allow you to store items in the cache with associated tags. Using tags creates internal hash tables that are not later removed even after the associated items are removed. This is not a memory leak, but it does contribute to the overall memory usage by the AppFabric Caching Service. If an application uses tags that change over time, this could contribute to memory pressure.

See Also

Concepts

Troubleshooting Server Issues

  2012-10-26