Free Trial *Internet Service Required
3 out of 6 rated this helpful - Rate this topic

Troubleshooting and Diagnostics for Windows Azure Caching

Windows Azure Caching applications are configured for diagnostic data collection using a diagnostic level. This diagnostic level provides a single setting for configuring the level of data collection. This topic provides an overview of troubleshooting steps and diagnostics for troubleshooting Windows Azure Caching applications.

Troubleshooting and Diagnostics for Windows Azure Caching Applications

Applications running on Windows Azure are widely distributed, multi-instance applications that can be hard to debug. These types of applications require more than standard tools and approaches to troubleshooting. Troubleshooting Windows Azure applications is beyond the scope of this topic, but more information is available in Troubleshooting in Windows Azure and Diagnostics and Debugging in Windows Azure. These topics discuss proven troubleshooting practices and contain links to more intensive information and best practices. For more information about viewing diagnostic data once it is collected, see Overview of Storing and Viewing Diagnostic Data in Windows Azure Storage.

Configuring Caching applications to collect diagnostic data for troubleshooting requires configuring the diagnostic level and configuring the cache diagnostics during role startup. This topic provides an overview of these steps for troubleshooting Windows Azure Caching applications.

Configure the Diagnostic Level

Windows Azure Caching provides five diagnostic levels which are used to configure the amount of diagnostic data collection. Diagnostic levels range from 0 to 4, and the number indicates the volume of diagnostic information collected for cache servers and clients. Each diagnostic level contains a pre-configured set of performance counters, event logs, tracing, and crash dump settings that can be used to monitor the health of your Caching application.

 

Diagnostic Level Diagnostic Data Collected

0

Very critical/catastrophic server logs only.

1

Diagnostic data that helps in assessing usage patterns, health of the cache system, and any potential errors. This is the default configuration level.

2

Diagnostic data at fine grain granularity of all requests and important system information.

3

Diagnostic data with more verboseness and system information.

4

Highest verbosity logs for all requests and system information.

There are two types of caching diagnostic levels that correspond to cache client and cache server diagnostics. DiagnosticLevel represents cache server diagnostics, and ClientDiagnosticLevel represents cache client diagnostics. Each level configures a different set of performance counters, logs, traces, and crash dump settings. These diagnostic levels are configured in the ServiceConfiguration.cscfg file for the application, and may be changed for a running cloud application simply by deploying an updated ServiceConfiguration.cscfg file.

Cache Server Diagnostic Levels

The cache server diagnostic level for a cache cluster is specified in the ServiceConfiguration.cscfg file, in the configuration section for the role that hosts the cache cluster. This setting is added by Visual Studio when caching is enabled in the Caching tab for the role, and is already present when a Cache Worker Role is added to a project. The default DiagnosticLevel is 1, and if no DiagnosticLevel is present for the role then a DiagnosticLevel of 1 is used. To change the DiagnosticLevel, modify the DiagnosticLevel setting for the role in ServiceConfiguration.cscfg. The following example is the WorkerRole1 section from a ServiceConfiguration.cscfg file with a DiagnosticLevel of 1.

Copy 
<Role name="WorkerRole1">
  <Instances count="1" />
  <ConfigurationSettings>
    <!-- Other settings omitted for clarity... -->
    <Setting name="Microsoft.WindowsAzure.Plugins.Caching.DiagnosticLevel" value="1" />
  </ConfigurationSettings>
</Role>

The following table lists the diagnostic data that is collected for each of the diagnostic levels.

 

DiagnosticLevel Data Collected

0
  • Log Level: Critical

  • No Performance Counters

  • No Event Logs registered

  • Mini Crash Dumps

1

2
  • Log Level: Information

  • Server Diagnostic Level 2 Performance Counters

  • The following event logs registered, in addition to the event logs from diagnostic level 1

    • Windows Azure!*

    • Microsoft-Windows-Application Server-System Services/Admin!*

    • Microsoft-Windows-Application Server-System Services/Operational!*

  • Full Crash Dumps

3

4
noteNote
When running Caching applications in the Windows Azure emulator, the traces for all role instances are collected under the *_IN_0 folder, even when there are multiple role instances. This behavior is by design. When Caching applications are hosted in Windows Azure, the traces for each role instance are located in the respective folders for the role instances.

Cache Client Diagnostic Levels

The ClientDiagnosticLevel level for a cache client is specified in the ServiceConfiguration.cscfg, in the configuration section for the role that is the cache client. This setting is added by the Windows Azure Caching NuGet package when the role is configured to act as a cache client.

noteNote
For more information about configuring cache clients using the Windows Azure Caching NuGet package, see Configure a Project for Caching on Roles.

The default ClientDiagnosticLevel is 1, and if no ClientDiagnosticLevel is present for the role then a ClientDiagnosticLevel of 1 is used. To change the ClientDiagnosticLevel, modify the ClientDiagnosticLevel setting for the role in ServiceConfiguration.cscfg. The following example is the WebRole1 section from a ServiceConfiguration.cscfg file with a ClientDiagnosticLevel of 1.

Copy 
<Role name="WebRole1">
  <Instances count="1" />
  <ConfigurationSettings>
    <!-- Other settings omitted for clarity... -->
    <Setting name="Microsoft.WindowsAzure.Plugins.Caching.ClientDiagnosticLevel" value="1" />
  </ConfigurationSettings>
</Role>

The following table lists the diagnostic data that is collected for each of the client diagnostic levels.

noteNote
Crash dump settings for the client diagnostic levels are relevant only to cache clients using the Memcached client shim. Crash dumps are not collected for non-Memcache clients. For more information about the Memcached client shim, see Memcached Wrapper for Windows Azure Caching.

 

ClientDiagnosticLevel Data Collected

0
  • Log Level: Critical

  • No Performance Counters

  • No Event Logs registered

  • Mini Crash Dumps

1

2

3

4
noteNote
When running Caching applications in the Windows Azure emulator, the traces for all role instances are collected under the *_IN_0 folder, even when there are multiple role instances. This behavior is by design. When Caching applications are hosted in Windows Azure, the traces for each role instance are located in the respective folders for the role instances.

Configure Cache Diagnostics during Role Startup

To start the collection of caching diagnostic data, the following code must be invoked during role startup. This code must be added to each role that is a cache client or cache server in order for caching diagnostic data to be collected. One convenient place to host this code is in the role's OnStart override, as demonstrated in the following example.

C#
VB
Copy 
public override bool OnStart()
{
    DiagnosticMonitorConfiguration dmConfig = 
        DiagnosticMonitor.GetDefaultInitialConfiguration();

    // Configure the collection of cache diagnostic data.
    CacheDiagnostics.ConfigureDiagnostics(dmConfig);

    DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", 
        dmConfig);

    return base.OnStart();
}

The ConfigureDiagnostics method configures the collection of caching diagnostic data. If this method is not called, caching diagnostic data is not collected. To use the CacheDiagnostics class, add a reference to Microsoft.ApplicationServer.Caching.AzureCommon.dll located in C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-10\ref\Caching, and add the following using (or Imports) statement.

C#
VB
Copy 
using Microsoft.ApplicationServer.Caching.AzureCommon;
noteNote
If the role has been configured as a cache client using the Caching NuGet package, then the Microsoft.ApplicationServer.Caching.AzureCommon.dll assembly reference is already added.

If you are adding caching to an existing role and diagnostic configuration code is already present, you can add the call to ConfigureDiagnostics to the existing diagnostic startup code before the call to Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitor.Start.

WarningWarning
Calling both ConfigureDiagnostics and Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitor.Start is required for diagnostic data collection in Windows Azure Caching applications. If both of these methods are not called during role startup, caching diagnostic data is not collected.

Performance Counters by Diagnostic Level

The following sections contain lists of the performance counters included in each of the server and client diagnostic levels. Both Caching and .NET Framework performance counters are included. For a complete list with descriptions of the Windows Azure Caching performance counters, see Monitor Windows Azure Caching. For more information about the .NET Framework performance counters, see Performance Counters in the .NET Framework.

Server Diagnostic Level 1 Performance Counters

DiagnosticLevel 1 contains the following performance counters.

  • \Windows Azure Caching:Host\Average Secondary Response Time / operation Microsecond

  • \Windows Azure Caching:Host\Total Data Size Bytes

  • \Windows Azure Caching:Host\Total Client Requests

  • \Windows Azure Caching:Host\Total Read Requests

  • \Windows Azure Caching:Host\Total Object Count

  • \Windows Azure Caching:Host\Total Failure Exceptions

  • \Memory\Available MBytes

  • \Process(CacheService)\Page Faults/sec

    • When running in the Windows Azure emulator the counter name is \Process(CacheServiceEmulator)\Page Faults/sec

  • \Processor(_Total)\% Processor Time

  • \.NET CLR Memory(_Global_)\% Time in GC

Server Diagnostic Level 2 Performance Counters

DiagnosticLevel 2 contains the following performance counters.

  • \Windows Azure Caching:Host\Average Secondary Response Time / operation Microsecond

  • \Windows Azure Caching:Host\Total Data Size Bytes

  • \Windows Azure Caching:Host\Total Client Requests

  • \Windows Azure Caching:Host\Total Read Requests

  • \Windows Azure Caching:Host\Total Object Count

  • \Windows Azure Caching:Host\Total Failure Exceptions

  • \Processor(_Total)\% Processor Time

  • \.NET CLR Memory(_Global_)\% Time in GC

  • \Windows Azure Caching:Host\Total Read Requests /sec

  • \Windows Azure Caching:Host\Cache Miss Percentage

  • \Windows Azure Caching:Host\Total Primary Data Size Bytes

  • \Windows Azure Caching:Host\Total Allocated Directory Count

  • \Windows Azure Caching:Host\Available Cache Item Percentage

  • \Windows Azure Caching:Host\Total Available Cache Item Count

  • \Windows Azure Caching:Host\Total Notification Delivered

  • \Windows Azure Caching:Host\Total Eviction Runs

  • \Windows Azure Caching:Host\Total Memory Evicted

  • \Windows Azure Caching:Host\Total Evicted Objects

  • \Windows Azure Caching:Host\Total Expired Objects

  • \Windows Azure Caching:Host\Total Requests Served

  • \Windows Azure Caching:Host\Total Write Operations

  • \.NET CLR LocksAndThreads(*)\Contention Rate / sec

  • \.NET CLR Memory(*)\% Time in GC

  • \.NET CLR Memory(*)\Gen 0 heap size

  • \.NET CLR Memory(*)\Gen 1 heap size

  • \.NET CLR Memory(*)\Gen 2 heap size

  • \.NET CLR Memory(*)\Large Object Heap size

  • \Memory\Available MBytes

  • \Process(CacheService)\Page Faults/sec

    • When running in the Windows Azure emulator the counter name is \Process(CacheServiceEmulator)\Page Faults/sec

  • \Process(*)\Thread Count

  • \Process(*)\Working Set

  • \Process(*)\Virtual Bytes

  • \Process(*)\Elapsed Time

  • \Process(*)\Page Faults/sec

  • \Network Interface(*)\Current Bandwidth

  • \Network Interface(*)\Bytes Total/sec

  • \Windows Azure Caching:Host\Total Connections Count

  • \Windows Azure Caching:Host\Average Quorum Response Time / operation Microsecond

  • \Windows Azure Caching:Host\Total Client Requests /sec

  • \Windows Azure Caching:Host\Total Notification Delivered

  • \Windows Azure Caching:Host\Total Get Misses

  • \Windows Azure Caching:Host\Total Write Operations /sec

Server Diagnostic Level 3 and 4 Performance Counters

DiagnosticLevel 3 and DiagnosticLevel 4 contain the following performance counters.

  • \.NET CLR LocksAndThreads(*)\Contention Rate / sec

  • \.NET CLR LocksAndThreads(*)\Current Queue Length

  • \.NET CLR Memory(*)\# Bytes in all Heaps

  • \.NET CLR Memory(*)\# Gen 0 Collections

  • \.NET CLR Memory(*)\# Gen 1 Collections

  • \.NET CLR Memory(*)\# Gen 2 Collections

  • \.NET CLR Memory(*)\% Time in GC

  • \.NET CLR Memory(*)\Gen 0 heap size

  • \.NET CLR Memory(*)\Gen 1 heap size

  • \.NET CLR Memory(*)\Gen 2 heap size

  • \.NET CLR Memory(*)\Large Object Heap size

  • \.NET CLR Exceptions(*)\# of Exceps Thrown

  • \Memory\Available MBytes

  • \Memory\Page Faults/sec

  • \PhysicalDisk(_Total)\% Idle Time

  • \PhysicalDisk(_Total)\Avg. Disk Queue Length

  • \Processor(_Total)\% Privileged Time

  • \Processor(_Total)\% Processor Time

  • \System\Context Switches/sec

  • \System\Processor Queue Length

  • \Process(*)\% Processor Time

  • \Process(*)\Handle Count

  • \Process(*)\Private Bytes

  • \Process(*)\Thread Count

  • \Process(*)\Working Set

  • \Process(*)\Virtual Bytes

  • \Process(*)\Elapsed Time

  • \Process(*)\Page Faults/sec

  • \Process(CacheService)\Page Faults/sec

    • When running in the Windows Azure emulator the counter name is \Process(CacheServiceEmulator)\Page Faults/sec

  • \Network Interface(*)\Current Bandwidth

  • \Network Interface(*)\Packets/sec

  • \Network Interface(*)\Bytes Sent/sec

  • \Network Interface(*)\Bytes Received/sec

  • \Network Interface(*)\Bytes Total/sec

  • \System\System Up Time

  • \Windows Azure Caching:Host\Available Cache Item Percentage

  • \Windows Azure Caching:Host\Total Connections Count

  • \Windows Azure Caching:Host\Available Directory Percentage

  • \Windows Azure Caching:Host\Available Memory Percentage

  • \Windows Azure Caching:Host\Average Quorum Response Time / operation Microsecond

  • \Windows Azure Caching:Host\Average Secondary Response Time / operation Microsecond

  • \Windows Azure Caching:Host\Cache Miss Percentage

  • \Windows Azure Caching:Host\Gateway Process Time

  • \Windows Azure Caching:Host\Gateway Failure Percentage

  • \Windows Azure Caching:Host\Request Processing Error Percentage

  • \Windows Azure Caching:Host\Total Allocated Cache Item Count

  • \Windows Azure Caching:Host\Total Allocated Directory Count

  • \Windows Azure Caching:Host\Total Available Cache Item Count

  • \Windows Azure Caching:Host\Total Available Directory Count

  • \Windows Azure Caching:Host\Total Available Memory Bytes

  • \Windows Azure Caching:Host\Total Cache Misses

  • \Windows Azure Caching:Host\Total Cache Misses /sec

  • \Windows Azure Caching:Host\Total Client Requests

  • \Windows Azure Caching:Host\Total Client Requests /sec

  • \Windows Azure Caching:Host\Total Data Size Bytes

  • \Windows Azure Caching:Host\Total Evicted Objects

  • \Windows Azure Caching:Host\Total Eviction Runs

  • \Windows Azure Caching:Host\Total Expired Objects

  • \Windows Azure Caching:Host\Total Failure Exceptions

  • \Windows Azure Caching:Host\Total Failure Exceptions /sec

  • \Windows Azure Caching:Host\Total Memory Evicted

  • \Windows Azure Caching:Host\Total Notification Delivered

  • \Windows Azure Caching:Host\Total Notification Delivered /sec

  • \Windows Azure Caching:Host\Total Notification Poll Requests

  • \Windows Azure Caching:Host\Total Notification Poll Requests /sec

  • \Windows Azure Caching:Host\Total Object Count

  • \Windows Azure Caching:Host\Total Objects Returned

  • \Windows Azure Caching:Host\Total Objects Returned /sec

  • \Windows Azure Caching:Host\Total Primary Data Size Bytes

  • \Windows Azure Caching:Host\Total Get Misses

  • \Windows Azure Caching:Host\Total Get Misses /sec

  • \Windows Azure Caching:Host\Total Get Requests

  • \Windows Azure Caching:Host\Total Get Requests /sec

  • \Windows Azure Caching:Host\Total GetAndLock Requests

  • \Windows Azure Caching:Host\Total GetAndLock Requests /sec

  • \Windows Azure Caching:Host\Total Memory Evicted

  • \Windows Azure Caching:Host\Total Read Requests

  • \Windows Azure Caching:Host\Total Read Requests /sec

  • \Windows Azure Caching:Host\Total Requests Served

  • \Windows Azure Caching:Host\Total Requests Served /sec

  • \Windows Azure Caching:Host\Total Retry Exception

  • \Windows Azure Caching:Host\Total Retry Exception /sec

  • \Windows Azure Caching:Host\Total Secondary Data Size Bytes

  • \Windows Azure Caching:Host\Total Successful GetAndLock Requests

  • \Windows Azure Caching:Host\Total Successful GetAndLock Requests /sec

  • \Windows Azure Caching:Host\Total Write Operations

  • \Windows Azure Caching:Host\Total Write Operations /sec

Client Diagnostic Level 1 Performance Counters

ClientDiagnosticLevel 1 contains the following performance counters.

  • \Windows Azure Caching:Client Host\Failure Exceptions

  • \Windows Azure Caching:Client Host\Total Local Cache Hits

  • \Windows Azure Caching:Client Host\Current Server Connections

  • \Windows Azure Caching:Client Host\Average Get Latency / operation Microsecond

  • \Windows Azure Caching:Client Host\Average Put Latency / operation Microsecond

  • \Windows Azure Caching:Client Host\Retry Exceptions

  • \Windows Azure Caching:Client Host\Timeout Exceptions

  • \Windows Azure Caching:Client Host\Requests

  • \Processor(_Total)\% Processor Time

  • \.NET CLR Memory(_Global_)\% Time in GC

  • \Windows Azure Caching:Client(*)\Failure Exceptions

  • \Windows Azure Caching:Client(*)\Total Local Cache Hits

  • \Windows Azure Caching:Client(*)\Current Server Connections

  • \Windows Azure Caching:Client(*)\Average Get Latency / operation Microsecond

  • \Windows Azure Caching:Client(*)\Average Put Latency / operation Microsecond

  • \Windows Azure Caching:Client(*)\Retry Exceptions

  • \Windows Azure Caching:Client(*)\Timeout Exceptions

  • \Windows Azure Caching:Client(*)\Requests

Client Diagnostic Level 2, 3, and 4 Performance Counters

ClientDiagnosticLevel 2, 3, and 4 contain the following performance counters.

  • \Windows Azure Caching:Client Host\Requests

  • \Windows Azure Caching:Client Host\Requests / sec

  • \Windows Azure Caching:Client Host\Server Responses Dropped / sec

  • \Windows Azure Caching:Client Host\Failure Exceptions

  • \Windows Azure Caching:Client Host\Failure Exceptions / sec

  • \Windows Azure Caching:Client Host\Average Get Latency / operation Microsecond

  • \Windows Azure Caching:Client Host\Average Put Latency / operation Microsecond

  • \Windows Azure Caching:Client Host\Average Get Latency (Network) / operation Microsecond

  • \Windows Azure Caching:Client Host\Read Requests

  • \Windows Azure Caching:Client Host\Write Requests

  • \Windows Azure Caching:Client Host\Bytes Received / sec

  • \Windows Azure Caching:Client Host\Bytes Sent / sec

  • \Windows Azure Caching:Client Host\Current Server Connections

  • \Windows Azure Caching:Client Host\Local Cache Filled Percentage

  • \Windows Azure Caching:Client Host\Local Cache Hits Percentage

  • \Windows Azure Caching:Client Host\Total Local Cache Hits

  • \Windows Azure Caching:Client Host\Total Local Cache Objects

  • \Windows Azure Caching:Client Host\Total Notifications Received

  • \Windows Azure Caching:Client Host\Timeout Exceptions

  • \Windows Azure Caching:Client Host\Timeout Exceptions / sec

  • \Windows Azure Caching:Client Host\Retry Exceptions

  • \Windows Azure Caching:Client Host\Retry Exceptions / sec

  • \Windows Azure Caching:Client Host\Total Connection Requests Failed

  • \Windows Azure Caching:Client Host\Network Exceptions

  • \Windows Azure Caching:Client Host\Network Exceptions / sec

  • \Windows Azure Caching:Client Host\Current Waiting Requests

  • \Processor(_Total)\% Processor Time

  • \.NET CLR Memory(_Global_)\% Time in GC

  • \Windows Azure Caching:Client(*)\Requests

  • \Windows Azure Caching:Client(*)\Requests / sec

  • \Windows Azure Caching:Client(*)\Server Responses Dropped / sec

  • \Windows Azure Caching:Client(*)\Failure Exceptions

  • \Windows Azure Caching:Client(*)\Failure Exceptions / sec

  • \Windows Azure Caching:Client(*)\Average Get Latency / operation Microsecond

  • \Windows Azure Caching:Client(*)\Average Put Latency / operation Microsecond

  • \Windows Azure Caching:Client(*)\Average Get Latency (Network) / operation Microsecond

  • \Windows Azure Caching:Client(*)\Read Requests

  • \Windows Azure Caching:Client(*)\Write Requests

  • \Windows Azure Caching:Client(*)\Bytes Received / sec

  • \Windows Azure Caching:Client(*)\Bytes Sent / sec

  • \Windows Azure Caching:Client(*)\Current Server Connections

  • \Windows Azure Caching:Client(*)\Local Cache Filled Percentage

  • \Windows Azure Caching:Client(*)\Local Cache Hits Percentage

  • \Windows Azure Caching:Client(*)\Total Local Cache Hits

  • \Windows Azure Caching:Client(*)\Total Local Cache Objects

  • \Windows Azure Caching:Client(*)\Total Notifications Received

  • \Windows Azure Caching:Client(*)\Timeout Exceptions

  • \Windows Azure Caching:Client(*)\Timeout Exceptions / sec

  • \Windows Azure Caching:Client(*)\Retry Exceptions

  • \Windows Azure Caching:Client(*)\Retry Exceptions / sec

  • \Windows Azure Caching:Client(*)\Total Connection Requests Failed

  • \Windows Azure Caching:Client(*)\Network Exceptions

  • \Windows Azure Caching:Client(*)\Network Exceptions / sec

  • \Windows Azure Caching:Client(*)\Current Waiting Requests

See Also


Build Date:

2012-10-26
Did you find this helpful?
(1500 characters remaining)

Community Additions

ADD
© 2013 Microsoft. All rights reserved.
facebook page visit twitter rss feed newsletter