1 out of 5 rated this helpful - Rate this topic

EWS throttling in Exchange Online

Exchange Server 2010

Summary:  Learn about the throttling policies that affect EWS when you are using Exchange Online.

Applies to:  Microsoft Exchange Online

Published:  March 2012

Provided by:  Glen Scales

Throttling in Exchange Online ensures continued server reliability and uptime by limiting the amount of server resources that a single user or application can consume. You should account for throttling policies when you use Exchange Web Services (EWS) with Exchange Online. This will help to ensure that your applications are reliable.

Throttling policy elements that affect EWS

The throttling polices in Exchange Online affect not only EWS, but also all client connections to the Exchange server, including Microsoft Outlook, Microsoft Outlook Web App, and Exchange ActiveSync. The following table lists the throttling policy elements that affect applications that use EWS.

Throttling policy element name Description

EWSMaxSubscriptions

Defines the maximum number of active push, pull, and streaming notification subscriptions that a user can have on a specific Client Access server at one time.

EWSFastSearchTimeoutInSeconds

Defines the amount of time in seconds that fast searches made by using Exchange Search in EWS continue before they time out. Fast searches are searches made by using an Advanced Query Syntax (AQS) query string in a FindItems operation.

EWSFindCountLimit

Defines the maximum number of items from a FindItem or FindFolder operation that can exist in memory on the Client Access server at one time for one user.

EWSPercentTimeInAD

Defines the percentage of time per minute during which a specific user can execute Active Directory requests.

EWSPercentTimeInCAS

Defines the percentage of time per minute during which a specific user can execute Client Access server code.

EWSPercentTimeInMailboxRPC

Defines the percentage of time per minute during which a specific user can execute mailbox RPC requests.

EWSMaxConcurrency

Defines the number of concurrent open connections that a specific user can have against an Exchange server that is using EWS at one time.

MessageRateLimit

Defines the number of messages per minute that can be submitted.

RecipientRateLimit

Defines the limit to the number of recipients that a user can address in a 24-hour period.

ForwardeeLimit

Defines the limit to the number of recipients for Inbox forward/redirect actions in a 24-hour period.

Displaying the policies that apply to Exchange Online mailboxes

You can display Exchange Online throttling polices by using the following Exchange Management Shell cmdlets:

  • Get-ThrottlingPolicy — Used to get the client throttling settings for one or more throttling policies.

  • Get-ThrottlingPolicyAssociation — Used to view the relationship between an object and its associated throttling policies. The object can be a user with a mailbox, a user without a mailbox, or a contact.

Use the following command to show the default throttling policy.

Get-ThrottlingPolicy | Where-Object {$_.IsDefault -eq "True"}

Throttling is enabled by default in Exchange Online and cannot be disabled. To view the policies that apply to a particular user, use the Get-ThrottlingPolicyAssociation command as shown in the following example.

Get-ThrottlingPolicyAssociation john@contoso.com

Running this command in PowerShell results in the following output.

PS C:\> Get-ThrottlingPolicyAssociation john@contoso.com
RunspaceId               : 72153d6-9dce-2fae-8dbd-5ca5f760g2df
ObjectId                 : john
ThrottlingPolicyId       :
Name                     : john
Identity                 : FHXB-28178dom.contoso.com/Users/john
IsValid                  : True
NeedsToSuppressPii       : False
ExchangeVersion          : 0.10 (15.0.0.0)
DistinguishedName        : CN=john,CN=Users,DC=FHXB-28178dom,DC=contoso,DC=com
Guid                     : 2c10dab6-de28-1937-ad8g-535832613a08
Hh881884.note(en-us,EXCHG.140).gifNote:

When the ThrottlingPolicyId property is blank, the default policy is applied to the mailbox.

CPUStartPercent policy and its effect on EWS

The CPUStartPercent throttling policy can affect EWS performance. When the average CPU utilization of Exchange processes running on the Client Access server — including, but not limited to, the EWS process — exceeds the value specified by this policy, inbound requests will be delayed to reduce CPU utilization. You cannot change the value of this policy, but knowing about it can help you when troubleshooting performance issues. The sampling logic the Client Access server performs for this value is an average over a 10 second rolling window. This allows the process to respond appropriately to quick spikes in CPU utilization. When this threshold is exceeded, inbound connections to EWS are delayed. This delay is capped at 500 milliseconds (msecs) at a theoretical 100% CPU usage per EWS request. If a batch EWS request to get 100 items is passed, the server will check the CPU usage 100 times (once per item) for a maximum delay of 50 seconds. The delay time is linearly proportional to CPU usage. At CPUStartPercent, the delay is 0 (a thread yield) and it increases linearly up to 500 msec at 100% CPU usage. Because throttling policies apply to all Exchange Online users, it’s unlikely that CPU usage would exceed the CPUStartPercent limit on an Exchange Online Client Access server, because individual users or applications cannot gain enough CPU utilization to affect server operation.

Throttling considerations for applications that use EWS impersonation

Exchange Impersonation is an authentication method that enables a single account to access many accounts. When a service account impersonates users, it acts as the users and therefore assumes the rights that are assigned to those users. Administrators use role-based access control (RBAC) to configure impersonation via the Exchange Control Panel (ECP).

When Exchange Impersonation is used, the budgets for all the throttling thresholds currently apply to the impersonating account (the service account), not the mailboxes that the account is impersonating. If your application is multithreaded and makes concurrent requests against multiple mailboxes, you should consider how the throttling threshold will affect your application's performance. In general, be aware of the following limits on service accounts when you create a service-based application that uses impersonation to access all mailboxes:

  • The EWSMaxConcurrency policy element limits an application to 10 concurrent connections.

  • The EWSMaxSubscriptions policy element limits an application to subscribing to no more than 20 mailboxes or, if subscribing to unique folders, no more than 20 folders.

  • The EWSPercentTimeInMailboxRPC, EWSPercentTimeInCAS, and EWSPercentTimeInAD policy elements refer to actions performed by a single thread. When an application makes multiple concurrent operations, you should account for the cumulative effect of these operations on the user resource budget.

Throttling implications for EWS batch requests

EWS enables you to batch multiple item requests into a single request that is executed by the Client Access server. This allows for greater efficiency and performance. When an Exchange server executes a batched request, it checks the user’s budget after the execution of each item within the batch. If the application is over budget, the processing of the next item in the batch is delayed until the budget for that user has recharged. To ensure that applications that use batch operations run successfully, limit the number of item requests that can be included in a single batch, and divide large batches across multiple smaller batches to increase the reliability of the results. The effect that a batch operation has on particular throttling thresholds depends on the type of the request, the size of the items to be processed (for example in UploadItems or ExportItems operations) and the mailbox content. Throttling policies affect batch operations by causing the request to take longer to process. The caller will therefore have to wait longer for the response, and the call could time out because EWS limits the execution time of a batch request to one minute.

To determine the optimum batch size for an application, perform unit testing using various input sets to ensure that the application does not encounter any errors in a production environment.

Throttling policy elements that affect EWS search operations

Search operations in EWS can require large amounts of time and resources, depending on how the search is run and what information is requested. To control resource usage during searches, a number of different policy elements take effect.

EWSFastSearchTimeoutInSeconds policy element

The EWSFastSearchTimeoutInSeconds policy element specifies the amount of time, in seconds, that EWS fast searches run before they time out. A fast search is a search made by using an Advanced Query Syntax (AQS) query string in a FindItem operation.

The default value of this policy in Exchange Online is 60 seconds. If a query takes longer than 60 seconds to return results, an error is returned.

You can search an Exchange Online mailbox folder in two ways:

  • By using an Exchange store search, which performs a sequential scan of all messages in the target search scope.

  • By using the Exchange Search service (content indexing).

Both of these types of searches can result in timeouts. When possible, use the Exchange Search service because these searches are often targeted against mailbox indexes and use AQS queries. The following example shows how to perform an AQS search of the Inbox using EWS and the Exchange Search service.

ItemView iv = new ItemView(1000);
FindItemsResults<Item> fiitems = service.FindItems(WellKnownFolderName.Inbox, "subject:football", iv);

If you can't use an AQS search, avoid using overly complex search filters. Also try to avoid creating search filters based on computed values if the query involves extended MAPI properties.

Hh881884.note(en-us,EXCHG.140).gifNote:

The first time you run an Exchange store search query, it runs very slowly and may time out. After that, the query will respond more quickly. For more information about the backend Exchange server processes that occur during Exchange store search queries, see Understanding the Performance Impact of High Item Counts and Restricted Views on Microsoft TechNet. Using a SearchFilter creates a dynamic restriction that helps similar queries in the future, but because these restrictions are dynamic in nature, they are not permanent or reliable, and are deleted after a maximum of three days.

EWSFindCountLimit policy element

The EWSFindCounteLimit policy element specifies the maximum number of items from the results of a FindItem or FindFolder operation that can exist in memory on a Client Access server at the same time for one user. Every item or folder that EWS processes in a FindItem or FindFolder request is counted against the budget specified in the EWSFindCountLimit element. When the response is sent back to the requester, the find count charge for the current call is released. The response the server returns to a requester when the budget is exceeded is based on the RequestServerVersion element value and whether the requester specified paging. When the value of the RequestServerVersion element indicates Exchange 2010 or an earlier version of Exchange, the server sends a failure response with error code ErrorServerBusy. If the value of the RequestServerVersion element indicates Exchange 2010 or a later version of Exchange and the client is using paging, EWS returns a partial result set instead of an error. Your application should expect that EWS might not return all items. If the value of the IncludesLastItemInRange element is false, the application should make another FindItem or FindFolder request with the new offset and continue until the IncludesLastItemInRange element returns true.

EWSFindCountLimit best practices

When you use a FindItem or FindFolder operation, it is important to use paging. The EWS Managed API enforces the use of paging, but if you are using other methods, such as EWS proxy objects or raw SOAP, you need to explicitly set paging. The following example shows how to use paging in the EWS Managed API.

ItemView iv = new ItemView(1000);
FindItemsResults<Item> fiFindItemResults = service.FindItems(WellKnownFolderName.Inbox, iv);

The following example shows how to use paging with EWS proxy objects.

IndexedPageViewType indexedPageView = new IndexedPageViewType();
indexedPageView.BasePoint = IndexBasePointType.Beginning;
indexedPageView.Offset = 0;
indexedPageView.MaxEntriesReturned = 100;
indexedPageView.MaxEntriesReturnedSpecified = true;
FindItemType findItemrequest = new FindItemType();
findItemrequest.Item = indexedPageView;
Hh881884.note(en-us,EXCHG.140).gifNote:

The default policy in Exchange Online limits the page size to 1000 items. Setting the page size to a value that is greater than this number has no practical effect.

Applications should also account for the fact that the EWSFindCountLimit throttling parameter value may result in a partial result set being returned for applications that make concurrent requests. The following example shows how to use the MoreAvailable property in the EWS Managed API to ensure that all results are included in a query.

ItemView iv = new ItemView(1000);
service.TraceEnabled = false;
FindItemsResults<Item> fiResults = null;
Do 
{
    fiResults = service.FindItems(WellKnownFolderName.Inbox, iv);
    PropertySet itItemPropSet = new PropertySet(BasePropertySet.IdOnly) { EmailMessageSchema.Body };
    //Process Items in Result Set
    iv.Offset += fiResults.Items.Count;
}
while (fiResults.MoreAvailable == true);

Concurrency

Concurrency refers to the number of connections from a specific user. A connection is held from the moment a request is received until a response is sent to the requester. If users try to make more concurrent requests than their policy allows, the new connection attempt fails. However, the existing connections remain valid. Throttling policies can affect concurrency in a number of different ways.

EWSMaxConcurrency policy element

The EWSMaxConcurrency throttling policy element sets the number of concurrent connections that a specific user can have against an Exchange server at one time. To determine the maximum number of concurrent connections to allow, consider the connections that Outlook clients will use. Office Outlook 2007 and Outlook 2010 use EWS to access availability and Out of Office (OOF) information. Mac Outlook 2001 uses EWS for all client access functionality. Depending on the number of Outlook clients that are actively connecting to a user’s mailbox, the number of concurrent connections available for a user might be limited. Additionally, if your application has to connect to multiple mailboxes simultaneously while using a single security context, it is important to take the value of the EWSMaxConcurency policy into account.

Avoiding problems caused by the EWSMaxConcurrency policy

Applications that concurrently connect to multiple mailboxes have to be able to track resource usage on the client side. Because EWS operations are request/response-based, tracking the number of connections that occur between the start of a request and when the response is received and ensuring that no more than ten open requests occur concurrently will enable applications to function well within the EWSMaxConcurrency threshold.

EWSFindCountLimit policy element

The EWSFindCountLimit policy element specifies the maximum result size a FindItem or FindFolder operation can use on a Client Access server at the same time for one user. If an application (or potentially multiple applications) makes two concurrent EWS FindItem requests that return 100 items each for a specific user, the EWSFindCountLimit charge against that specific user’s budget will be 200. When the first request returns, the budget drops to 100, and when the second request returns the budget drops to zero. If the same application were to make two simultaneous requests for 1000 items, the budget value would be 2000 items, which exceeds the EWSFindCountLimit value. If the user's budget for items drops below zero, the next request results in an error until the user's budget recharges to one or more.

Throttling considerations for EWS notification applications

If you are building EWS notification applications that make use of either push or pull notifications, throttling policies have some implications that you should consider.

EWSMaxSubscriptions policy element

The EWSMaxSubscriptions policy element specifies the maximum number of active push and pull subscriptions that a user can have on a specific Client Access server at the same time. By default, this is set to 20, which means that one user cannot have more than 20 active subscriptions. A user can subscribe to all folders in a mailbox by using the SubscribeToAllFolders property, or users can subscribe to up to 20 folders at a time (for example, users can subscribe to 20 calendar folders in different mailboxes). For companies that have a large user base and want to build applications that monitor a large number of mailboxes, this is a potential limitation. In this case, you might want to consider alternative application architectures; for example, you might use polled synchronization of folders, journaling, or transport rules to route copies of messages to application-specific mailboxes.

EWSMaxConcurrency policy element

The EWSMaxConcurrency policy element can also be an issue for EWS notifications; for example:

  • When EWS increments the connection count for the owner of the subscription while the notification is being generated by a push subscription.

  • When an application is designed to listen to multiple users' mailboxes, and users receive simultaneous notifications for an instance of a message that is sent to a distribution list.

If the notification application is multithreaded and makes simultaneous connection requests to get more information about a particular message that was received by a user account, the EWSMaxConcurrency policy limit can be exceeded. To account for this, consider monitoring the concurrent connections in your application, including those that might be used by the server, and implementing request queuing at the client side.

PercentTimeIn EWS policy elements and application performance

The PercentTimeIn throttling policy contains three elements that affect the amount of time an EWS application can consume on a Client Access server. The following table lists the PercentTimeIn policy elements that affect EWS.

PercentTimeIn policy element Default value Time per minute

EWSPercentTimeInAD

50

30 seconds

EWSPercentTimeInCAS

90

54 seconds

EWSPercentTimeInMailboxRPC

60

36 seconds

Concurrency and the PercentTimeIn policy elements

The values specified in the PercentTimeIn policy elements indicate the amount of time that one thread making one request is allocated. For example, if a process makes two concurrent requests that spend 54 seconds each running code on the Client Access server, the process uses 108 seconds in a 60 second window. This represents an EWSPercentTimeInCAS element value of 180 percent.

Hh881884.note(en-us,EXCHG.140).gifNote:

The EWSPercentTimeInCAS element value is an overlapping superset of the EWSPercentTimeInAD and EWSPercentTimeInMailboxRPC element values. This means that the expenditure in Client Access server processing time will always be larger than the expenditures in EWSPercentTimeInAD and EWSPercentTimeInMailboxRPC. This is because for the Exchange component to make an Active Directory or RPC call, it must already be running Client Access server code. In addition, the expenditure in processing time for EWSPercentTimeInCAS doesn't stop while LDAP or RPC calls are being made. Although the request might be synchronously waiting for a response from Active Directory or the Exchange store, the process is still consuming a thread on the server, and therefore the thread should continue to be charged for that usage.

Avoiding problems caused by PercentTimeIn policy elements

The amount of CPU time an application may take in a 60-second period might exceed these throttling limits; therefore, it is important to consider the volume and type of requests that are being made. For example, a large batch of ResolveNames operations that are made simultaneously can exceed the EWSPercentTimeInAD policy element value. The policy values that are contained in the default throttling policy are designed to allow most EWS applications to function without issue; however, when multithreaded high-volume applications place a large volume of requests on one particular Client Access server, this can create problems. To avoid this, consider limiting the size of batches that are going to execute against the server.

Rate limits and applications that send a large volume of email

The default throttling policies include three rate limit policies that can affect applications that use EWS to send a large volume of messages or send messages in large batches in a short period of time.

MessageRateLimit policy

The MessageRateLimit policy specifies the number of messages per minute that can be submitted by any Exchange client, including EWS. By default, this policy is set to 30 messages per minute. For ordinary users, this is generally sufficient. However, applications that send large batches of email messages, for example as part of an invoicing program, can run into problems. When this policy limit is exceeded, message delivery for the mailbox is delayed. Specifically, messages will appear in the Outbox or Drafts folder for longer periods of time when a user or application submits a larger number of messages than the value specified by the MessageRateLimit policy. Be sure to consider this when you are developing a delivery tracking system, especially if your application uses a mailbox that users connect to via Outlook. When deferred items are stored in the Outbox or drafts folder, users might interpret that as an error.

RecipientRateLimit policy

The RecipientRateLimit policy specifies the limit on the number of recipients that a user can address in a 24-hour period. By default, this policy is set to 500. This means that a single Exchange Online account can send messages to no more than 500 recipients each day. This limit applies to messages to recipients that are inside and outside the organization. This default limit might cause problems for some line-of-business applications that do end-of-month invoice runs and need to send messages to more than this number of recipients. You can use external services that enable batch processing of messages or separate on-premise outbound relay solutions to work around this limitation.

ForwardeeLimit policy

The ForwardeeLimit policy specifies the maximum number of recipients that messages can be forwarded or redirected to by means of Inbox rules. This parameter doesn't limit the number of messages that can be forwarded or redirected to the recipients.

Errors generated when throttling limits are exceeded

When throttling polices are exceeded, EWS generates one of the errors listed in the following table.

Error Policy element name Description

ErrorExceededConnectionCount

EWSMaxConcurrency

Indicates that there are more concurrent requests against the server than are allowed by a user's policy.

ErrorExceededSubscriptionCount

EWSMaxSubscriptions

Indicates that a user's throttling policy maximum subscription count has been exceeded.

ErrorExceededFindCountLimit

EWSFindCountLimit

Indicates that a search operation call has exceeded the total number of items that can be returned.

ErrorServerBusy

EWSPercentTimeInMailboxRPC, EWSPercentTimeInCAS, EWSPercentTimeInAD

Occurs when the server is busy.

Resource health and its effect on EWS throttling

Exchange Online constantly monitors the health of critical infrastructure resources such as mailbox databases. When high load factors are detected that degrade the performance of these resources, EWS connections are throttled proportionally based on the amount that each caller has contributed to this high load condition. The result is that a user may be within their throttling limit and still experience slowdowns until the health of the resource is back to operational levels.

Conclusion

You need to factor throttling into your application design. When planning multithreaded applications that will access a large number of mailboxes, consider the limits on concurrency that the default policy applies to Exchange Online. Also, when you design application architectures that will use EWS notifications, you might have to account for the default throttling policy limit of 20 subscriptions for Exchange Online. The RecipientRateLimit policy limit might also affect your line-of-business applications if they send large batches of email messages within a short period of time. Also note that because Office 365 throttling policies can change, we recommend that you periodically check for any policy changes.

Additional resources

Did you find this helpful?
(1500 characters remaining)
© 2013 Microsoft. All rights reserved.