Windows Azure Queues and Windows Azure Service Bus Queues

Windows Azure supports two types of queue mechanisms: Windows Azure Queues and Service Bus Queues.

Windows Azure Queues, which are part of the Windows Azure storage infrastructure, feature a simple REST-based Get/Put/Peek interface, providing reliable, persistent messaging within and between services.

Service Bus Queues are part of a broader Windows Azure messaging infrastructure that supports queuing as well as publish/subscribe, Web service remoting, and integration patterns.

While both queuing technologies exist concurrently, Windows Azure Queues were introduced first, as a dedicated queue storage mechanism built on top of the Windows Azure storage services. Service Bus Queues, introduced with the latest release of the Service Bus, are built on top of the broader “brokered messaging” infrastructure designed to integrate applications or application components that may span multiple communication protocols, data contracts, trust domains, and/or network environments.

This article compares the two queue technologies offered by Windows Azure by discussing the differences in the behavior and implementation of the features provided by each. The article also provides guidance for choosing which features might best suit your application development needs.

Both Windows Azure Queues and Service Bus Queues are implementations of the message queuing service currently offered on Windows Azure. Each has a slightly different feature set, which means you can choose one or the other, or use both, depending on the needs of your particular solution or business/technical problem you are solving.

When determining which queuing technology fits the purpose for a given solution, solution architects and developers should consider the recommendations below. Further details can be found in the next section.

As a solution architect/developer, you should consider using Windows Azure Queues when:

Your application needs to store over 5 GB worth of messages in a queue, where the messages have a lifetime shorter than 7 days.
Your application requires flexible leasing to process its messages. This allows messages to have a very short lease time, so that if a worker crashes, the message can be processed again quickly. It also allows a worker to extend the lease on a message if it needs more time to process it, which helps deal with non-deterministic processing time of messages.
Your application wants to track progress for processing a message inside of the message. This is useful if the worker processing a message crashes. A subsequent worker can then use that information to continue where the prior worker left off.
You require server side logs of all of the transactions executed against your queues.

As a solution architect/developer, you should consider using queues in the Windows Azure Service Bus when:

You require full integration with the Windows Communication Foundation (WCF) communication stack in the .NET Framework.
Your solution needs to be able to support automatic duplicate detection.
You need to be able to process related messages as a single logical group.
Your solution requires transactional behavior and atomicity when sending or receiving multiple messages from a queue.
The time-to-live (TTL) characteristic of the application-specific workload can exceed the 7-day period.
Your application handles messages that can exceed 64 KB but will not likely approach the 256 KB limit.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
Your solution must be able to receive messages without having to poll the queue. With the Service Bus, this can be achieved through the use of the long-polling receive operation.
You deal with a requirement to provide a role-based access model to the queues, and different rights/permissions for senders and receivers.
Your queue size will not grow larger than 5 GB.
You can envision an eventual migration from queue-based point-to-point communication to a message exchange pattern that allows seamless integration of additional receivers (subscribers), each of which receives independent copies of either some or all messages sent to the queue. The latter refers to the publish/subscribe capability natively provided by the Service Bus.
Your messaging solution needs to be able to support the “At-Most-Once” delivery guarantee without the need for you to build the additional infrastructure components.
You would like to be able to publish and consume message batches.

The tables in the following sections provide a logical grouping of queue features and let you compare, at a glance, the capabilities available in both Windows Azure Queues and Windows Azure Service Bus Queues.

This section compares some of the fundamental queuing capabilities provided by Windows Azure Queues and Service Bus Queues.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Ordering guarantee	No	Yes - First-In-First-Out (FIFO) (through the use of messaging sessions)
Delivery guarantee	At-Least-Once	At-Least-Once At-Most-Once
Transaction support	No	Yes (through the use of local transactions)
Receive behavior	Non-blocking (completes immediately if no new message is found)	Blocking with/without timeout (offers long polling, or the “Comet technique”) Non-blocking (through the use of .NET managed API only)
Receive mode	Peek & Lease	Peek & Lock Receive & Delete
Exclusive access mode	Lease-based	Lock-based
Lease/Lock duration	30 seconds (default) 7 days (maximum)	60 seconds (default) You can renew a message lock using the RenewLock API.
Lease/Lock granularity	Message level (each message can have a different timeout value)	Queue level (each queue has a lock granularity applied to all of its messages, but you can renew the lock using the RenewLock API.)
Batched receive	Yes (explicitly specifying message count when retrieving messages, up to a maximum of 32 messages)	Yes (implicitly enabling a pre-fetch property or explicitly through the use of transactions)
Batched send	No	Yes (through the use of transactions or client-side batching)

Additional Information

The guaranteed FIFO pattern in Service Bus Queues requires the use of messaging sessions. In the event that the application crashes while processing a message received in the Peek & Lock mode, the next time a queue receiver accepts a messaging session, it will start with the failed message after its time-to-live (TTL) period expires.
Service Bus Queues support the At-Least-Once delivery guarantee. In addition, the At-Most-Once semantic can be supported by using session state to store the application state and by using transactions to atomically receive messages and update the session state. The Windows Azure Workflow Service uses this technique to guarantee At-Most-Once delivery.
Service Bus Queues provide support for local transactions in the context of a single queue.
The Receive and Delete mode supported by the Service Bus provides the ability to reduce the messaging operation count (and associated cost) in exchange for lowered delivery assurance.
Windows Azure Queues provide leases with the ability to extend the leases for messages. This allows the workers to maintain short leases on messages. Thus, if a worker crashes, the message can be quickly processed again by another worker. In addition, a worker can extend the lease on a message if it needs to process it longer than the current lease time.
Windows Azure Queues offer a visibility timeout that you can set upon the enqueueing or dequeuing of a message. In addition, you can update a message with different lease values at runtime, and update different values across messages in the same queue. Service Bus lock timeouts are defined in the queue metadata, however you can renew the lock by calling the RenewLock method.
The maximum timeout for a blocking receive operation in Service Bus Queues is 24 days.
Client-side batching provided by the Service Bus allows a queue client to batch multiple messages into a single send operation. Batching is only available for asynchronous send operations.

This section compares advanced capabilities provided by Windows Azure Queues and Service Bus Queues.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Scheduled delivery	Yes	Yes
Automatic dead lettering	No	Yes
Message deferral	Yes (via in-place update of visibility timeout)	Yes (provided via a dedicated API function)
Poison message support	Yes	Yes
In-place update	Yes	No
Server-side transaction log	Yes	No
Storage metrics	Yes	No
Purge queue function	Yes	No
Message groups	No	Yes (through the use of messaging sessions)
Duplicate detection	No	Yes (configurable on the sender side)
WCF integration	No	Yes (offers out-of-the-box WCF bindings)
WF integration	Custom (requires building a custom WF activity)	Native (offers out-of-the-box WF activities)

Additional Information

As of the November 2011 release of the Windows Azure SDK, both queuing technologies allow a message to be scheduled for delivery at a later time.
Windows Azure Queues provide support for updating message content. This functionality can be used for persisting state information and incremental progress updates into the message so that it can be processed from the last known checkpoint, instead of starting from scratch.
Dead lettering, which is only supported by Service Bus Queues, can be useful for isolating messages that cannot be processed successfully by the receiving application or when messages cannot reach their destination due to an expired time-to-live (TTL) property. The TTL value specifies how long a message remains in the queue. With the Service Bus, the message will be moved to a special queue called the $DeadLetterQueue when the TTL period expires.
To find “poison” messages in Windows Azure Queues, when dequeuing a message the application examines the DequeueCount property of the message. If DequeueCount is above a given threshold, the application moves the message to an application-defined “dead letter” queue.
Windows Azure Queues enable you to obtain a detailed log of all of the transactions executed against the queue, as well as aggregated metrics. Both of these are useful for debugging and understanding how your application uses Windows Azure Queues. They are also useful for performance-tuning your application and reducing the costs of using queues.
The concept of “message sessions” supported by the Service Bus allows messages that belong to a certain logical group to be associated with a given receiver, which in turn creates a session-like affinity between messages and their respective receivers. You can enable this advanced functionality in the Service Bus by setting the SessionID property on a message. Receivers can then listen on a specific session ID and receive messages that share the specified session identifier.
The duplication detection functionality supported by Service Bus Queues automatically removes duplicate messages sent to a queue or topic, based on the value of the MessageID property.

This section compares Windows Azure Queues and Service Bus Queues from the perspective of capacity and quotas that may apply.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Maximum message size	64 KB (48 KB when using Base64 encoding)	256 KB (including both header and body, maximum header size: 64 KB)
Maximum queue size	100 TB (limited to a single storage account capacity)	1, 2, 3, 4 or 5 GB (defined upon creation of a queue)
Maximum message TTL	7 days	Unlimited
Maximum number of queues	Unlimited	10,000 (per service namespace, can be increased)
Maximum number of concurrent clients	Unlimited	Unlimited (100 concurrent connection limit only applies to TCP protocol-based communication)

Additional Information

With Windows Azure Queues, if the content of the message is not XML-safe, then it must be Base64 encoded. If you Base64-encode the message, the user payload can be up to 48 KB, instead of 64 KB.
With Service Bus Queues, each message stored in a queue is comprised of two parts: a header and a body. The total size of the message cannot exceed 256 KB.
When clients communicate with Service Bus Queues over the TCP protocol, the maximum number of concurrent connections to a single Service Bus Queue is limited to 100. This number is shared between senders and receivers. If this quota is reached, subsequent requests for additional connections will be rejected and an exception will be received by the calling code. This limit is not imposed on clients connecting to the queues using REST-based API.
The Service Bus enforces queue size limits. The maximum queue size is specified upon creation of the queue and can have a value of 1, 2, 3, 4, or 5 GB. If the queue size value set on creation of the queue is reached, additional incoming messages will be rejected and an exception will be received by the calling code. For more information about quotas in the Service Bus, see Windows Azure Service Bus Quotas.
If you require more than 10,000 queues in a single Service Bus service namespace, you can contact the Windows Azure support team and request an increase. To scale beyond 10,000 queues with the Service Bus, you can also create additional service namespaces using the Windows Azure Management Portal.

This section compares the management features provided by Windows Azure Queues and Service Bus Queues.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Management protocol	REST over HTTP/HTTPS	REST over HTTPS
Runtime protocol	REST over HTTP/HTTPS	REST over HTTPS TCP with TLS
.NET Managed API	Yes (.NET managed Storage Client API)	Yes (.NET managed brokered messaging API)
Java API	Yes	Yes
PHP API	Yes	Yes
Node.js API	Yes	Yes
Arbitrary metadata support	Yes	No
Queue naming rules	Up to 63 characters long (letters in a queue name must be lowercase)	Up to 260 characters long (queue names are case-insensitive)
Get queue length function	Yes (approximate value)	Yes (exact, point-in-time value)
Peek function	Yes	No

Additional Information

Windows Azure Queues provide support for arbitrary attributes that can be applied to the queue description, in the form of name/value pairs.
Windows Azure Queues also offer the ability to peek a message without having to lock it, which can be useful when implementing a queue explorer/browser tool.
The Service Bus .NET managed brokered messaging API leverages full-duplex TCP connections for improved performance when compared to REST over HTTP.
Windows Azure Queues impose certain restrictions on queue names. For more information, see the Naming Queues and Metadata article.
Service Bus Queue names can be up to 260 characters long and have less restrictive naming rules. Service Bus Queue names can contain only letters, numbers, periods (.), hyphens (-) and underscores (_).

This section compares Windows Azure Queues and Service Bus Queues from a performance perspective.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Maximum throughput	Up to 2,000 messages per second	Up to 2,000 messages per second (based on benchmark with 1 KB messages)
Average latency	10 ms (with TCP Nagle disabled)	100 ms
Throttling behavior	Reject with HTTP 503 code (throttled requests are not treated as billable)	Reject with exception/HTTP 503 (throttled requests are not treated as billable)

Comparison Criteria

Windows Azure Queues

Service Bus Queues

Maximum throughput

Up to 2,000 messages per second

(based on benchmark with 1 KB messages)

Average latency

10 ms

(with TCP Nagle disabled)

100 ms

Throttling behavior

Reject with HTTP 503 code

(throttled requests are not treated as billable)

Reject with exception/HTTP 503

(throttled requests are not treated as billable)

Additional Information

A single Windows Azure Queue can process up to 2,000 transactions per second. A transaction is either a Put, Get, or Delete operation. Sending a single message to a queue (Put) is counted as one transaction, but receiving a message is often a two-step process involving the retrieval (Get), followed by a request to remove the message from the queue (Delete). As a result, a successful dequeue operation usually involves two transactions.
When your application reaches the maximum throughput for a Windows Azure Queue, an “HTTP 503 Server Busy” response is usually returned from the queue service. When this occurs, the application should trigger the retry logic with exponential back-off delay.
The latency of Windows Azure Queues is 10 milliseconds on average when handling small messages (less than 10 KB) from a hosted service located in the same location (sub-region) as the storage account.
Both Windows Azure Queues and Service Bus Queues enforce throttling behavior by rejecting requests to a queue that is being throttled. However, neither of them treats throttled requests as billable.
Benchmarks against Service Bus Queues have demonstrated that a single queue can achieve a message throughput of up to 2,000 messages per second with a message size of approximately 1 KB. To achieve higher throughput, use multiple queues. For more information about performance optimization with the Service Bus, see Best Practices for Performance Improvements Using Service Bus Brokered Messaging.
When the maximum throughput is reached for a Service Bus Queue, a ServerBusyException (when using the.NET managed brokered messaging API) or HTTP 503 (when using the REST-based API) response will be returned to the queue client, indicating that the queue is being throttled.

This section discusses the authentication and authorization features supported by Windows Azure Queues and Service Bus Queues.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Authentication	Symmetric key	ACS claims
Role-based access control	No	Yes (through the use of ACS roles)
Identity provider federation	No	Yes

Comparison Criteria

Windows Azure Queues

Service Bus Queues

Authentication

Symmetric key

ACS claims

Role-based access control

No

Yes

(through the use of ACS roles)

Identity provider federation

No

Yes

Additional Information

Every request to either of the queuing technologies must be authenticated. Public queues with anonymous access are not supported.
The authentication scheme provided by Windows Azure Queues involves the use of a symmetric key, which is a hash-based Message Authentication Code (HMAC), computed with the SHA-256 algorithm and encoded as a Base64 string. For more details about the respective protocol, refer to the Authenticating Access to Your Storage Account article.
The Windows Azure Active Directory Access Control (also known as Access Control Service or ACS) supported by the Service Bus offers three distinct roles: Admin, Sender, and Receiver, which is not supported at this time for Windows Azure Queues.
Because the Service Bus offers ACS integration, it enables you to federate with Active Directory (through the use of ADFS) and common web identity providers such as Live ID, Google, Facebook, and Yahoo.

This section compares Windows Azure Queues and Service Bus Queues from a cost perspective.

Comparison Criteria	Windows Azure Queues	Service Bus Queues
Queue transaction cost	$0.01 (per 10,000 transactions)	$0.01 (per 10,000 messages)
Billable operations	All	Send/Receive Only (no charge for other operations)
Idle transactions	Billable (querying an empty queue is counted as a billable transaction)	Billable (a receive against an empty queue is considered a billable message)
Storage cost	$0.14 (per GB/month)	$0.00
Outbound data transfer costs	$0.12 - $0.19 (depending on geography)	$0.12 - $0.19 (depending on geography)
Windows Azure Active Directory Access Control (also known as Access Control Service or ACS) transaction costs	$0.00 (as ACS is not being used)	$1.99 (per 100,000 token requests, see comment below)

Additional Information

Data transfers are charged based on the total amount of data leaving the Windows Azure datacenters via the Internet in a given billing period.
Data transfers between Windows Azure services located within the same sub-region are not subject to charge.
As of this writing, all inbound data transfers are not subject to charge.
The cost of ACS transactions is insignificant when performing messaging operations against Service Bus Queues. The Service Bus acquires one ACS token per a single instance of the messaging factory object. The token is then reused until it expires, after about 20 minutes. Therefore, the volume of messaging operations in the Service Bus is not directly proportional to the amount of ACS transactions required to support these operations.
Given the support for long polling, using Service Bus Queues can be cost effective in situations where low-latency delivery is required.

Note
All costs are subject to change. The table above reflects current pricing as of the writing of this article and does not include any promotional offers that may currently be available. For up-to-date pricing information, see the Pricing Overview page.

The decision on when to use Windows Azure Queues or Service Bus Queues clearly depends on a number of factors. These factors may depend heavily on the individual needs of your application and its architecture. If your application already uses the core capabilities of Windows Azure, you may prefer to choose Windows Azure Queues, especially if you require basic communication and messaging between services or need queues that can be larger than 5 GB in size.

Because Service Bus Queues provide a number of advanced features, such as sessions, transactions, duplicate detection, automatic dead-lettering, and durable publish/subscribe capabilities, they may be a preferred choice if you are building a hybrid application or if your application otherwise requires these features.

This article began with a summary of prescriptive recommendations, then listed and compared the features of each queuing technology offered by Windows Azure today. By listing features in functional groups, the topic presented a visually compelling side-by-side analysis that should help you understand the similarities and differences between Windows Azure Queues and Service Bus Queues.

By gaining a deeper understanding of the two technologies, you will be able to make a more informed decision on which queue technology to use, and when.

Best Practices for Leveraging Windows Azure Service Bus Brokered Messaging API

Best Practices for Performance Improvements Using Service Bus Brokered Messaging

How to Use Service Bus Queues

How to Use the Queue Storage Service

Introducing Queues and Topics in Azure Service Bus

The Developer's Guide to Service Bus

“Windows Azure Tables and Queues Deep Dive”

Windows Azure Storage Architecture

“Using the Queuing Service in Windows Azure”

Understanding Windows Azure Storage Billing – Bandwidth, Transactions, and Capacity