Storage Infrastructure

Summary: This article discusses the basics of Windows Azure Storage. It explores the architecture of the storage system, and explains how it provides available, durable, reliable and highly-scalable storage services.

Windows Azure Storage includes a robust set of services. The services are always available, durable, and highly scalable. They form an integral part of a cloud-enabled application. These benefits are the result of a complex architecture that offers many options. Understanding this architecture can be daunting, and you may wonder where to start. This article explores the basic concepts and technologies that comprise the Windows Azure storage system. It then focuses on partitioning, and explains why partitions are critical to scalable applications.

The storage infrastructure for Windows Azure is comprised of subscriptions, storage accounts, and storage services. Subscriptions are the entry points into Windows Azure itself. Without it, you cannot access the management tools that you need to create the accounts and services. Subscriptions are currently associated with a Windows Live ID. To acquire a subscription, explore the various offers and pricing models. See http://www.microsoft.com/windowsazure/offers/ for more information. After you create a subscription, you can access the Windows Azure Portal site (https://manage.windowsazure.com). From here, you can create storage accounts, as well as other accounts, such as hosting accounts for your Compute services. Storage accounts are discussed in more detail in the next section. The various storage services, such as blobs, tables and queues are also discussed in more detail in the Abstractions section.

Each subscription can have zero or more storage accounts. Storage accounts are the entry points into your storage services. Each storage account has a one hundred terabyte capacity. To create a storage account, log into the Windows Azure Management Portal and navigate to Hosted Services, Storage Accounts & CDN view. Click the New Storage Account button to display the following dialog box.

The value you enter in the URL field is the name of your storage account. This name must be unique among all existing storage services because it forms the first part of the service's hostname. The second part of the hostname indicates the type of service it is. For example, if the storage account name is "mystorageaccount123", then the blob service uses the mystorageaccount123.blob.core.windows.net hostname. Next, you need to specify the region from which the storage account will be serviced. The region indicates the geographic location of the data center that will host your services. You should choose the region based on where your users are located. If you plan to deploy a Windows Azure application that will use your storage account, the region for both the storage account and the hosted service should be the same. Choosing the same region provides cost benefits. For example, the bandwidth between the storage services and the hosted services is free, and there will be ample bandwidth, and low latency periods. These are the different regions that you can currently select:

• US North Central
  
  
• US South Central
  
  
• Europe North
  
  
• Europe West
  
  
• Asia West
  
  
• Asia Southeast
  
  

The following figure illustrates the general location of each region.

When you build an application that uses one or more storage services, you can develop it against local instances of those services. The storage emulator is an application that provides local blob, table and queue services in your development environment. The storage emulator is part of the Windows Azure Software Development Kit (SDK), which can be downloaded from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=7A1089B6-4050-4307-86C4-9DADAA5ED018&displaylang=en. The storage emulator can help you to reduce the cost of development because you do not need to access the actual Windows Azure storage services, for which you are billed.

Unlike the Windows Azure storage services, which use a domain name, the local services that are hosted by the storage emulator are addressed with your local IP address of 127.0.0.1, and ports that range from 10000 to 10002. Here are the services and their addresses:

• Blob - http://127.0.0.1:10000/devstoreaccount1/
  
  
• Queue - http://127.0.0.1:10001/devstoreaccount1/
  
  
• Table - http://127.0.0.1:10002/devstoreaccount1/
  
  

You will notice that the path in the address includes "devstoreaccount1". This is the account name for the emulated storage service. The storage emulator only supports one account. The account key that supplements the devstoreaccount1 account name is:

Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

The account name and key are well-known values for the storage emulator. Using account keys to access storage services is discussed in more detail in the Security section. You may also notice that the addresses use the HTTP protocol. The storage emulator does not support HTTPS.

There are several ways to start the storage emulator. The taskbar may be the easiest way. On the taskbar, click Start, All Programs, Windows Azure SDK v1.4, and then click Storage Emulator. This starts both the storage emulator, and the csmonitor application. The csmonitor application allows you to access the Storage Emulator UI. The following image shows the csmonitor menu.

The following image shows the Storage Emulator UI.

Another way to launch the storage emulator is by using the DSInit command-line tool. The tool can be run from the Windows Azure SDK Command Prompt. On the taskbar, click Start, All Programs, Windows Azure SDK v1.4, and then click Windows Azure SDK Command Prompt. The DSInit command accepts the following parameters:

• sqlInstance - This parameter provides the name of the SQL Server instance. You can use "." to indicate that you want to use the default instance. When you use SQL Express as your database, you must provide the name of the instance, such as "SQLEXPRESS."
  
  
• forceCreate - This parameter reinitializes the database.
  
  

The following image shows an example of how to use the DSInit command on a computer that runs SQL Express, which has an instance named SQLEXPRESS.

The following figure illustrates the status of the storage emulator.

Note
The command-line tool is useful when you want to automatically start the local, emulated services every time you restart your computer.

There are differences between the storage emulator and Windows Azure Storage. You may need to consider them to see which is better suited to your development effort. Here are the most important differences.

• The local, emulated services are not scalable and will not support a large number of clients.
  
  
• The local, emulated Blob service is limited to sizes of up to two gigabytes (2 GB).
  
  
• The local, emulated Table service only supports the range that is supported by SQL Server 2005. Additionally, it has the precision of SQL Server 2005, which is precise only up to 1/300th of a second.
  
  
• Rows in the local, emulated Table service are limited to less than 1 MB.
  
  

For the full list of differences between the storage emulator and Windows Azure Storage, see "Differences Between the Storage Emulator and Windows Azure Storage Services" at http://msdn.microsoft.com/en-us/library/gg433135.aspx.

Windows Azure storage services can be accessed from other services that run within Windows Azure or from the internet. The data that the services provide is secured by credentials. These credentials are comprised of the storage account name, and an access key. The storage account name does not usually change. However, the access key can be regenerated. The access key is a long base-64 string. Windows Azure provides you with two access keys. You can use either of the two access keys but it is recommended that you use the primary access key. The purpose of the secondary access key is to avoid downtime for your application when you do need to regenerate your access keys.

To manage your services, use your Windows Live ID to log into the portal. When you use the portal, you only need the Windows Live ID to access your services and manage them. However, you can also manage your services by using the set of management APIs. To use the management APIs, you must upload a certificate through the Windows Azure Portal. The certificate can be self-signed. The following figure illustrates a list of management certificates. Notice the buttons at the top that allow you to upload new certificates and delete existing ones.

After you upload a certificate, you can call any of the management APIs, although the call must include the certificate's public key. The APIs are functionally equivalent to the management tools that you can access through the portal. For example, if your application requires that new worker role instances start when your queue is backlogged, you can use the management APIs to reconfigure the current configuration with a new instance count. For more information on how to use the management APIs to dynamically scale a Windows Azure application, please read the Real World: Dynamically Scaling a Windows Azure Application.