Configuring Diagnostics for Azure Cloud Services
Updated: August 14, 2014
When you troubleshoot an Azure cloud service, you can configure Azure Diagnostics more easily by using Visual Studio. Azure Diagnostics captures system data and logging data on the virtual machine instances that run your cloud service and brings that data to your storage account. You can also access Azure Diagnostics programmatically and by editing configuration files directly. See Collect Logging Data by Using Azure Diagnostics.
You can configure Azure Diagnostics in the following ways:
Programmatically, in either code or a role, or from code that's running in a separate configuration application.
From the role designer in Visual Studio or by directly modifying the configuration files. The configuration file, diagnostics.wadcfg, is added to your project.
If you take either of these approaches, your changes will take effect the next time that you deploy the cloud service to Azure or run the service in the emulator.
By using Server Explorer to change the diagnostics settings for a running cloud service.
By using Visual Studio, you can customize the diagnostics data that you collect for a role that runs in Azure. When you change diagnostics settings in Visual Studio, you change the configuration file (diagnostics.wadcfg), so that the new settings are reflected when you next deploy your cloud service.
On the shortcut menu for the role that interests you, choose Properties, and then choose the Configuration tab.
In the Diagnostics section, make sure that the Enable Diagnostics check box is selected.
Of the available options (Errors only, All information, and Custom plan), Errors only is the default option and requires the least amount of storage because it doesn’t transfer warnings or tracing messages. All information transfers the most information and is, therefore, the most expensive option.
To customize the settings, choose the Custom plan option button, and then choose the Edit button.
The Diagnostics configuration dialog box, which contains tabs for each source of diagnostic data that you can collect, appears.
Specify the buffer size and transfer period for application logs.
By setting the buffer size on each tab, you reserve that amount of file system storage for each type of data on each virtual machine. The combined size of all data buffers can't exceed the overall diagnostics quota. In the lower-left corner of the dialog box, the running total of buffer sizes adjusts according to your choices.
Warning If you're investigating a cloud service that was built by using version 1.8 or earlier of the Azure SDK, you must set aside 500 MB of the available local storage for the garbage collector. The local storage value is stored in the
LocalStorageelement in the service definition (.csdef) file in your project. Later versions of the SDK have no such restriction.
Your application generates application logs by using the System.Diagnostics API. To generate data in these logs from your application code, add a reference to System.Diagnostics.dll, and write data by using the static methods of the Trace class, such as TraceError and TraceInformation.
Set the log level to one of the following values (in order from least information to most): Critical, Error, Warning, Information, or Verbose.
On the Event Logs tab, select the check boxes for the types of events that you want to track.
The categories correspond to the filters in the Windows Event Viewer.
On the Performance counters tab, select the check boxes for the performance counters that you want to track.
To track a performance counter that isn’t listed, enter it by using the suggested syntax. The operating system on the virtual machine determines which performance counters you can track. For more information about syntax, see Specifying a Counter Path.
On the Infrastructure Logs tab, specify the settings that you want.
These logs relate to the infrastructure of Azure Diagnostics.
On the Log Directories tab, configure the data that's collected from log directories for Internet Information Services (IIS) requests and crash dumps.
Deploy your cloud service as usual, and then run it.
You can view the diagnostics data in either a report that Visual Studio generates or tables in your storage account. To view the data in a report, open Server Explorer, and open the shortcut menu of the node for the role that interests you, and then choose View Diagnostic Data.
A report that shows the available data appears.
If the most recent data doesn't appear, you might have to wait for the transfer period to elapse.
Choose the Refresh button to update the data in real time.
In Server Explorer, open the storage account that's associated with the deployment.
Open the diagnostics tables in the table viewer, and then review the data that you collected. For IIS logs and custom logs, you can open a blob container. By reviewing the following table, you can find the table or blob container that contains the data that interests you. In addition to the data for that log file, the table entries contain EventTickCount, DeploymentId, Role, and RoleInstance to help you identify what virtual machine and role generated the data and when.
Diagnostic data Description Location
Logs that your code generates by calling methods of the System.Diagnostics.Trace class.
This data is from the Windows event logs on the virtual machines. Windows stores information in these logs, but applications and services also use them to report errors or log information.
You can collect data on any performance counter that’s available on the virtual machine. The operating system provides performance counters, which include many statistics such as memory usage and processor time.
These logs are generated from the diagnostics infrastructure itself.
These logs record web requests. If your cloud service gets a significant amount of traffic, these logs can be quite lengthy, so you should collect and store this data only when you need it.
You can find failed-request logs in the blob container under wad-iis-failedreqlogs under a path for that deployment, role, and instance. You can find complete logs under wad-iis-logfiles. Entries for each file are made in the WADDirectories table.
This information provides binary images of your cloud service’s process (typically a worker role).
wad-crush-dumps blob container
Custom log files
You can specify in code the location of custom log files in your storage account. For example, you can specify a custom blob container.
If data of any type is truncated, you can try increasing the buffer for that data type or shortening the interval between transfers of data from the virtual machine to your storage account.
(optional) Purge data from the storage account occasionally to reduce overall storage costs.
When you do a full deployment, the .wadcfg file is updated in Azure, and your cloud service picks up any changes to your diagnostics configuration. If you, instead, update an existing deployment, the .wadcfg file isn’t updated in Azure. You can still change diagnostics settings, though, by following the steps in the next section. For more information about performing a full deployment and updating an existing deployment, see Publish Azure Application Wizard.
If you're investigating a problem with a running cloud service, you might want to collect data that you didn't specify before you originally deployed the role. In this case, you can start to collect that data by using the settings in Server Explorer. You can configure diagnostics for either a single instance or all the instances in a role, depending on whether you open the Diagnostics Configuration dialog box from the shortcut menu for the instance or the role. If you configure the role node, any changes apply to all instances. If you configure the instance node, any changes apply to that instance only.
In Server Explorer, expand the Cloud Services node, and then expand nodes to locate the role or instance that you want to investigate or both.
On the shortcut menu for an instance node or a role node, choose Update Diagnostics Settings, and then choose the diagnostic settings that you want to collect.
For information about the various settings, see the previous procedure in this topic.
On the shortcut menu for the role or instance, choose View Diagnostics Data.
A report shows the available data.
If the most recent data don't appear, you might have to wait for the transfer period to elapse.
Choose the Refresh button to update the data in real time. You can also open the storage account that your cloud service uses, and watch for the data that you collected to appear in tables or blob containers. You might have to refresh the tables to see the most recent data.
If you change data collection in Server Explorer, these changes persist until you perform a full redeployment of your cloud service. If you use the default publish settings, the changes are not overwritten, since the default publish setting is to update the existing deployment, rather than do a full redeployment. To make sure the settings clear at deployment time, go to the Advanced Settings tab in the Publish Wizard, and clear the Deployment update checkbox. When you redeploy with that checkbox cleared, the settings revert to those in the .wadcfg file (as set through the Properties editor for the role). If you update your deployment, Azure keeps the old settings.
What if I have competing diagnostics configuration in code, in an application from outside my organization, or in the diagnostics.wadcfg file? Which information is used?
The diagnostics data that you set in the role designer, before deployment, changes the diagnostics.wadcfg file. You can further customize the collection of data by editing that file directly. Azure copies this configuration file to a Azure blob resource that's named wad-control-container under the deploymentID, role name, and role instance. Azure loads this configuration file from the blob container when your cloud service starts its diagnostic monitor. You can also use the APIs for Azure Diagnostics to control diagnostics in code, which overrides the configuration settings in diagnostics.wadcfg. However, these runtime changes are lost whenever a role restarts. You can also affect diagnostic configuration at runtime by using Server Explorer or writing your own utility that manages diagnostics configuration by using the diagnostic management APIs directly. Because these changes affect the configuration file in the blob container, they persist when a role restarts. For information about how to use each type of configuration and their precedence, see Collect Logging Data by Using Azure Diagnostics.
What is the buffer size, and how large should it be?
On each virtual machine instance, quotas limit how much diagnostic data can be stored on the local file system. In addition, you specify a buffer size for each type of diagnostic data that's available. This buffer size acts like an individual quota for that type of data. By checking the bottom of the dialog box, you can determine the overall quota and the amount of memory that remains. If you specify larger buffers or more types of data, you'll approach the overall quota. You can change the overall quota by modifying the diagnostics.wadcfg configuration file. The diagnostics data is stored on the same filesystem as your application’s data, so if your application uses a lot of disk space, you shouldn’t increase the overall diagnostics quota.
What is the transfer period, and how long should it be?
The transfer period is the amount of time that elapses between data captures. After each transfer period, data is moved from the local filesystem on a virtual machine to tables in your storage account. If the amount of data that's collected exceeds the quota before the end of a transfer period, older data is discarded. You might want to decrease the transfer period if you're losing data because your data exceeds the buffer size or the overall quota.
What time zone are the time stamps in?
The time stamps are in the local time zone of the data center that hosts your cloud service.
How do I manage costs when collecting diagnostic information?
The default settings (Log level set to Error and transfer period set to 1 minute) are designed to minimize cost. Your compute costs will increase if you collect more diagnostic data or decrease the transfer period. Don’t collect more data than you need, and don’t forget to disable data collection when you no longer need it. You can always enable it again, even at runtime, as the previous section shows.
How do I collect failed-request logs from IIS?
By default, IIS doesn’t collect failed-request logs. You can configure IIS to collect them if you edit the web.config file for your web role.
I’m not getting trace information from RoleEntryPoint methods like OnStart. What’s wrong?
The methods of RoleEntryPoint are called in the context of WAIISHost.exe, not IIS. Therefore, the configuration information in web.config that normally enables tracing doesn’t apply. To resolve this issue, add a.config file to your web role project, and name the file to match the output assembly that contains the RoleEntryPoint code. In the default web role project, the name of the .config file would be WAIISHost.exe.config. Then add the following lines to this file:
<system.diagnostics> <trace> <listeners> <add name “AzureDiagnostics” type=”Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener”> <filter type=”” /> </add> </listeners> </trace> </system.diagnostics>
Now, in the Properties window, set the Copy to Output Directory property to Copy always.