Monitoring StreamInsight Performance Counters and Events

 

You can now use your existing diagnostic infrastructure to monitor and troubleshoot StreamInsight processes and queries with Performance Monitor and Event Viewer.

StreamInsight installs and configures performance counters. You can monitor these counters with Performance Monitor or another monitoring tool.

Performance counters are available for the following StreamInsight objects:

  • Servers

  • Queries

  • Input Adapters

  • Resiliency

Configuring performance counters

The performance counters are installed and configured when you install StreamInsight. This setup option is not configurable.

Counters in the Servers category are on by default and cannot be turned off. Counters in the Queries and Input Adapters categories are off by default. When you turn on counters for a query, then counters for the associated input adapters are turned on also. When you restart an instance of StreamInsight, these default behaviors are restored.

How to turn on counters for a single query

You turn on counters for a specific query and its associated input adapters by enabling a new Aspect in a call to Server.SetDiagnosticSettings.

For example, you can use the following code to turn on the performance counters for the query with the Uri, cep:/Server/Application/MyApp/Query/MyQuery.

DiagnosticSettings settings = server.GetDiagnosticSettings(new Uri(“cep:/Server/Application/MyApp/Query/MyQuery”);  
Settings.Aspects |= DiagnosticAspect.PerformanceCounters;  
server.SetDiagnosticSettings(new Uri(“cep:/Server/Application/MyApp/Query/MyQuery”), settings);  
  

You can use the following code to turn off the performance counters for the same query.

DiagnosticSettings settings = server.GetDiagnosticSettings(new Uri(“cep:/Server/Application/MyApp/Query/MyQuery”);  
Settings.Aspects & ~DiagnosticAspect.PerformanceCounters;  
server.SetDiagnosticSettings(new Uri(“cep:/Server/Application/MyApp/Query/MyQuery”), settings);  
  

How to turn on counters for all queries

You can also use a foreach loop to turn on the performance counters for all the queries and input adapters in an application.

foreach (var q in application.Queries)  
   {  
       DiagnosticSettings settings = server.GetDiagnosticSettings(q.Value.Name);  
       settings.Aspects |= DiagnosticAspect.PerformanceCounters;  
       server.SetDiagnosticSettings(q.Value.Name, settings);  
   }  

For more information about the classes and methods used in the preceding code samples, see the following topics:

List of performance counters

Naming conventions

Strings such as application names that are longer than 16 characters are shortened according to the following formula: first 10 characters + 2 periods + last 4 characters.

In the specific naming conventions described below, uniqueid refers to the ID of the object at run time.

Server counters

Naming convention for server instances: %PROCESSNAME%[uniqueid], where %PROCESSNAME% is the name of the hosting process.

NameDescription
Events in input queuesNumber of events in input queues of all queries.
Events in output queuesNumber of events in output queues of all queries.
MemoryTotal memory (in bytes) used for all streams, events, and operator indexes.
Running queriesNumber of queries currently running.

Query counters

Naming convention for query instances: %APPLICATION NAME%,%QUERY NAME%[uniqueid],%SERVER_INSTANCE_NAME%, where %SERVER_INSTANCE_NAME% is the instance name of the server to which the query belongs.

NameDescription
Average produced event latencyThe average latency (in milliseconds) of events produced by the query each second.
CTIs producedNumber of CTIs produced.
Events in output queueNumber of events in output queue.
Events producedNumber of events produced.
MemoryTotal memory (in bytes) used, including events, indexes, and streams.
Produced events/secNumber of events produced per second.

Input Adapter counters

Naming convention for adapter instances: %APPLICATION NAME%,%QUERY NAME%,%ADAPTER NAME%[uniqueid],%SERVER_INSTANCE_NAME%.

NameDescription
Adjusted eventsNumber of events whose temporal metadata was adjusted according to the AdvanceTimeSettings policy.
CTIs inputNumber of CTIs sent to input queue.
Dropped eventsNumber of events dropped according to the AdvanceTimeSettings policy.
Events in input queueNumber of events in input queue (including CTIs).
Incoming events/secNumber of events received by the adapter per second.
Resumes/secNumber of state transitions into the running state per second.
Suspensions/secNumber of state transitions into the suspended state per second.
Total events enqueuedNumber of events enqueued.

How can performance counters help me?

Here are some examples of the scenarios in which performance counters can help you to understand, monitor, and troubleshoot your StreamInsight applications:

  • Server scenarios

    • Memory footprint. How much memory is the embedded instance of StreamInsight, separately from the hosting application?

    • Capacity planning. How many queries can I run on this server before performance degrades?

    • Post-mortem analysis. How many queries were running when the server crashed?

  • Query scenarios

    • Throughput monitoring. Where is the bottleneck?

    • Latency monitoring.

StreamInsight logs events to the Windows Application event log. You can review this log with Event Viewer or another monitoring tool. The logged events help you to monitor state changes and troubleshoot anomalies that may occur in your StreamInsight applications.

Events are logged by the following StreamInsight objects:

  • Server

  • Query

Configuring administrative logging

Administrative logging is on by default. You can turn administrative logging off and on only at the level of the StreamInsight instance (cep:/Server/).

How to turn off administrative logging

DiagnosticSettings settings = server.GetDiagnosticSettings(new Uri("cep:/Server/"));  
settings.Aspects &= ~DiagnosticAspect.AdminLog;  
server.SetDiagnosticSettings(new Uri("cep:/Server/"), settings);  

How to turn on administrative logging

DiagnosticSettings settings = server.GetDiagnosticSettings(new Uri("cep:/Server/"));  
settings.Aspects |= DiagnosticAspect.AdminLog;  
server.SetDiagnosticSettings(new Uri("cep:/Server/"), settings);  

List of administrative events

The following table lists the events that you may see in the event log.

The Source for all StreamInsight administrative events is StreamInsight.

Query events

SymbolEvent IDLevelMessageParameters
QueryInitializing11001InformationQuery {0} is initializing.Query Uri
QueryRunning11002InformationQuery {0} is running.Query Uri
QueryCheckpointing11003InformationQuery [0} is checkpointing.Query Uri
QueryStopping11004InformationQuery {0} is stopping.Query Uri
QuerySuspended11005InformationQuery {0} is suspended.Query Uri
QueryCompleted11006InformationQuery {0} is completed.Query Uri
QueryAborted11007WarningQuery {0} is aborted.Query Uri
QueryStopped11008InformationQuery {0} is stopped.Query Uri
QueryError11500WarningQuery {0} encountered an exception. Exception details: {1}Query Uri, exception message.
QueryRecoveryError11501ErrorQuery {0} encountered an exception during recovery. Exception details: {1}Query Uri, exception message.
QueryCheckpointError11502ErrorQuery {0} encountered an exception during checkpointing. Exception details: {1}Query Uri, exception message.

Server events

SymbolEvent IDLevelMessageParameters
ServerCreated1000InformationServer created.
ServerDisposed1001InformationServer disposed.
Show: