Export (0) Print
Expand All

Ingesting Assets in Bulk with the Media Services SDK for .NET

Updated: February 13, 2014

Uploading large asset files can be a bottleneck during asset creation. Ingesting Assets in Bulk or “Bulk Ingesting”, involves decoupling asset creation from the upload process. To use a bulk ingesting approach, create a manifest (IngestManifest) that describes the asset and its associated files. Then use the upload method of your choice to upload the associated files to the manifest’s blob container. Microsoft Azure Media Services watches the blob container associated with the manifest. Once a file is uploaded to the blob container, Microsoft Azure Media Services completes the asset creation based on the configuration of the asset in the manifest (IngestManifestAsset).

This topic describes ingesting assets in bulk using the Microsoft Azure Media Services SDK for .NET. For information on ingesting assets in bulk using REST API, see Ingesting Assets in Bulk with the REST API.

noteNote
Media Services uses the value of the IAssetFile.Name property when building URLs for the streaming content (for example, http://{WAMSAccount}.origin.mediaservices.windows.net/{GUID}/{IAssetFile.Name}/streamingParameters.) For this reason, percent-encoding is not allowed. The value of the Name property cannot have any of the following percent-encoding-reserved characters: !*'();:@&=+$,/?%#[]". Also, there can only be one ‘.’ for the file name extension.

ImportantImportant
If you have not yet downloaded and set up this project, see the directions in the introductory topic titled Building Applications with the Media Services SDK for .NET. Each code example in this topic is illustrated with one or more methods in the downloaded project (the solution file for the project is MediaServicesSDKSamples.sln). To find the code for a specific example, search for the example method name in the downloaded project. To run a specific code example, call the associated method with the required parameters.

The basic workflow for bulk ingesting is divided into the following sections:

noteNote
Developing applications with the Microsoft Azure Media Services SDK requires that you must first have a reference to the Media Services server context as described in the topic Connecting to Media Services with the Media Services SDK. The context object is represented by the variable named context in the following code examples.

Create the Ingest Manifest

The IngestManifest represents a set of assets to be created through bulk ingesting along with their associated asset files. To create a new IngestManifest call the Create method exposed by the IngestManifests collection on the server context. This method will create a new IngestManifest with the manifest name you provide.

Full example code is provided at the end of this topic in the Example Code section.

IIngestManifest manifest = context.IngestManifests.Create(name);

Create the Assets

Create the assets that will be associated with the bulk IngestManifest. These assets are created using the Assets collection on the server context. Configure the desired encryption options on the asset for bulk ingesting.

Full example code is provided at the end of this topic in the Example Code section.

// Create the assets that will be associated with this bulk ingest manifest
IAsset destAsset1 = _context.Assets.Create(name + "_asset_1", AssetCreationOptions.None);
IAsset destAsset2 = _context.Assets.Create(name + "_asset_2", AssetCreationOptions.None);

Create the IngestManifestAssets

An IngestManifestAsset associates an Asset with a bulk IngestManifest for bulk ingesting. It also associates the AssetFiles that will make up each Asset. To create an IngestManifestAsset, use the Create method on the server context.

The following example demonstrates adding two new IngestManifestAssets that associate the two assets previously created to the bulk ingest manifest. Each IngestManifestAsset also associates a set of files that will be uploaded for each asset during bulk ingesting. The file paths are part of the MediaServiceSDKSamples project.

Full example code is provided at the end of this topic in the Example Code section.


string filename1 = _singleInputMp4Path;
string filename2 = _primaryFilePath;
string filename3 = _singleInputFilePath;

IIngestManifestAsset bulkAsset1 =  manifest.IngestManifestAssets.Create(destAsset1, new[] { filename1 });
IIngestManifestAsset bulkAsset2 =  manifest.IngestManifestAssets.Create(destAsset2, new[] { filename2, filename3 });

Upload the Asset Files

You can use any high speed client application capable of uploading the asset files to the blob storage container URI provided by the BlobStorageUriForUpload property of the IngestManifest. One notable high speed upload service is Aspera On Demand for Azure Application. You can also write code to upload the assets files as shown in the following code example.

static void UploadBlobFile(string destBlobURI, string filename)
{
    Task copytask = new Task(() =>
    {
        var storageaccount = new CloudStorageAccount(new StorageCredentials(_storageAccountName, _storageAccountKey), true);
        CloudBlobClient blobClient = storageaccount.CreateCloudBlobClient();
        CloudBlobContainer blobContainer = blobClient.GetContainerReference(destBlobURI);

        string[] splitfilename = filename.Split('\\');
        var blob = blobContainer.GetBlockBlobReference(splitfilename[splitfilename.Length - 1]);

        using (var stream = System.IO.File.OpenRead(filename))
            blob.UploadFromStream(stream);

        lock (consoleWriteLock)
        {
            Console.WriteLine("Upload for {0} completed.", filename);
        }
    });

    copytask.Start();
}

The code for uploading the asset files for the sample used in this topic is shown in the following code example.


UploadBlobFile(manifest.BlobStorageUriForUpload, filename1);
UploadBlobFile(manifest.BlobStorageUriForUpload, filename2);
UploadBlobFile(manifest.BlobStorageUriForUpload, filename3);

Monitor the IngestManifest Progress

You can determine the progress of the bulk ingesting for all assets associated with an IngestManifest by polling the Statistics property of the IngestManifest. In order to update progress information, you must use a new server context each time you poll the Statistics property.

The following example demonstrates polling an IngestManifest by its Id.

Full example code is provided at the end of this topic in the Example Code section.

static void MonitorBulkManifest(string manifestID)
{
   bool bContinue = true;
   while (bContinue)
   {
      CloudMediaContext context = GetContext();
      IIngestManifest manifest = context.IngestManifests.Where(m => m.Id == manifestID).FirstOrDefault();

      if (manifest != null)
      {
         lock(consoleWriteLock)
         {
            Console.WriteLine("\nWaiting on all file uploads.");
            Console.WriteLine("PendingFilesCount  : {0}", manifest.Statistics.PendingFilesCount);
            Console.WriteLine("FinishedFilesCount : {0}", manifest.Statistics.FinishedFilesCount);
            Console.WriteLine("{0}% complete.\n", (float)manifest.Statistics.FinishedFilesCount / (float)(manifest.Statistics.FinishedFilesCount + manifest.Statistics.PendingFilesCount) * 100);

            if (manifest.Statistics.PendingFilesCount == 0)
            {
               Console.WriteLine("Completed\n");
               bContinue = false;
            }
         }

         if (manifest.Statistics.FinishedFilesCount < manifest.Statistics.PendingFilesCount)
            Thread.Sleep(60000);
      }
      else //=== Manifest is null ===//
         bContinue = false;
   }
}

Example Code

The following example code is derived from the MediaServicesSDKSample project which is available for download here: Building Applications with the Media Services SDK for .NET.

using System;
using System.Configuration;
using System.Globalization;
using System.IO;
using System.Net;
using System.Runtime.Serialization.Json;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
using System.Linq;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.MediaServices.Client;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Auth;
using System.Collections.Generic;
using System.Reflection;

namespace Microsoft.Samples.WindowsAzureMediaServicesSDK.BulkIngest
{
    class Program
    {
        private static CloudMediaContext _context = null;
        private static readonly string _supportFiles = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location) + @"\..\..\supportFiles";
        private static readonly string _singleInputFilePath =
            Path.GetFullPath(_supportFiles + @"\multifile\interview2.wmv");
        private static readonly string _singleInputMp4Path =
            Path.GetFullPath(_supportFiles + @"\multifile\BigBuckBunny.mp4");
        private static readonly string _primaryFilePath =
            Path.GetFullPath(_supportFiles + @"\multifile\interview1.wmv");

        private static System.Object consoleWriteLock = new Object();

        private static readonly string _accountName =
            ConfigurationManager.AppSettings["MediaServicesAccountName"];
        private static readonly string _accountKey =
            ConfigurationManager.AppSettings["MediaServicesAccountKey"];

        // Media Services storage account credentials.
        private static readonly string _storageAccountName =
            ConfigurationManager.AppSettings["MediaServicesStorageAccountName"];
        private static readonly string _storageAccountKey =
            ConfigurationManager.AppSettings["MediaServicesStorageAccountKey"];

        static void Main(string[] args)
        {
            _context = GetContext();
            string manifestName = "TestManifest";

            CreateBulkIngestManifest(manifestName);
            ListIngestManifests();
            DeleteBulkManifest(manifestName);
        }


        static CloudMediaContext GetContext()
        {
            return new CloudMediaContext(_accountName, _accountKey);
        }

        static IIngestManifest CreateBulkIngestManifest(string name)
        {
            Console.WriteLine("\n===============================================");
            Console.WriteLine("========[ CREATE BULK INGEST MANIFEST ]========");
            Console.WriteLine("===============================================\n");


            IIngestManifest manifest = _context.IngestManifests.Create(name);

            IAsset destAsset1 = _context.Assets.Create(name + "_asset_1", AssetCreationOptions.None);
            IAsset destAsset2 = _context.Assets.Create(name + "_asset_2", AssetCreationOptions.None);

            string filename1 = _singleInputMp4Path;
            string filename2 = _primaryFilePath;
            string filename3 = _singleInputFilePath;

            //=== Preently, each asset filename uploaded must be unique for an individual Bulk ingest manifest. So two assets can not have ===//
            //=== the same asset filename or an exception will be thrown for duplicate filename.                                           ===//
            IIngestManifestAsset bulkAsset1 = manifest.IngestManifestAssets.Create(destAsset1, new[] { filename1 });
            IIngestManifestAsset bulkAsset2 = manifest.IngestManifestAssets.Create(destAsset2, new[] { filename2, filename3 });

            ListIngestManifests(manifest.Id);

            Console.WriteLine("\n===============================================");
            Console.WriteLine("===[ BULK INGEST MANIFEST MONITOR FILE COPY]===");
            Console.WriteLine("===============================================\n");

            UploadBlobFile(manifest.BlobStorageUriForUpload, filename1);
            UploadBlobFile(manifest.BlobStorageUriForUpload, filename2);
            UploadBlobFile(manifest.BlobStorageUriForUpload, filename3);

            MonitorBulkManifest(manifest.Id);
            ListIngestManifests(manifest.Id);

            return manifest;
        }


        // Upload a file into Blob Storage
        static void UploadBlobFile(string destBlobURI, string filename)
        {
            Task copytask = new Task(() =>
            {
                var storageaccount = new CloudStorageAccount(new StorageCredentials(_storageAccountName, _storageAccountKey), true);
                CloudBlobClient blobClient = storageaccount.CreateCloudBlobClient();
                CloudBlobContainer blobContainer = blobClient.GetContainerReference(destBlobURI);

                string[] splitfilename = filename.Split('\\');
                var blob = blobContainer.GetBlockBlobReference(splitfilename[splitfilename.Length - 1]);

                using (var stream = System.IO.File.OpenRead(filename))
                    blob.UploadFromStream(stream);

                lock (consoleWriteLock)
                {
                    Console.WriteLine("Upload for {0} completed.", filename);
                }
            });

            copytask.Start();
        }

        static void DeleteBulkManifest(string name)
        {
            Console.WriteLine("\n===============================================");
            Console.WriteLine("=======[ DELETE BULK INGEST MANIFESTS ]========");
            Console.WriteLine("===============================================\n");

            var manifest = _context.IngestManifests.Where(c => c.Name == name).FirstOrDefault();
            DeleteBulkManifestAssets(manifest.Id);

            Console.WriteLine("Deleting Manifest...\n\tName : {0}\n\tManifest ID : {1}...", manifest.Name, manifest.Id);
            manifest.Delete();
            Console.WriteLine("Delete Complete.\n");
        }

        static void DeleteBulkManifestAssets(string manifestID)
        {
            Console.WriteLine("\n===============================================");
            Console.WriteLine("=====[ DELETE BULK INGEST MANIFEST ASSETS ]====");
            Console.WriteLine("===============================================\n");

            foreach (IIngestManifest manifest in _context.IngestManifests.Where(c => c.Id == manifestID))
            {
                Console.WriteLine("Deleting assets for manifest named : {0}...\n", manifest.Name);
                foreach (IIngestManifestAsset manifestAsset in manifest.IngestManifestAssets)
                {
                    foreach (ILocator locator in manifestAsset.Asset.Locators)
                    {
                        Console.WriteLine("Deleting locator {0} for asset {1}", locator.Path, manifestAsset.Asset.Id);
                        locator.Delete();
                    }
                    Console.WriteLine("Deleting asset {0}\n", manifestAsset.Asset.Id);
                    manifestAsset.Asset.Delete();
                }
            }
        }

        static void MonitorBulkManifest(string manifestID)
        {
            bool bContinue = true;
            while (bContinue)
            {
                //=== We need a new context here because IIngestManifestStatistics is considered an expensive ===//
                //=== property and not updated realtime for a context.                                        ===//
                CloudMediaContext context = GetContext();

                IIngestManifest manifest = context.IngestManifests.Where(m => m.Id == manifestID).FirstOrDefault();

                if (manifest != null)
                {
                    lock (consoleWriteLock)
                    {
                        Console.WriteLine("\nWaiting on all file uploads.");
                        Console.WriteLine("PendingFilesCount  : {0}", manifest.Statistics.PendingFilesCount);
                        Console.WriteLine("FinishedFilesCount : {0}", manifest.Statistics.FinishedFilesCount);
                        Console.WriteLine("{0}% complete.\n", (float)manifest.Statistics.FinishedFilesCount / (float)(manifest.Statistics.FinishedFilesCount + manifest.Statistics.PendingFilesCount) * 100);


                        if (manifest.Statistics.PendingFilesCount == 0)
                        {
                            Console.WriteLine("Completed\n");
                            bContinue = false;
                        }
                    }

                    if (manifest.Statistics.FinishedFilesCount < manifest.Statistics.PendingFilesCount)
                    {
                        Thread.Sleep(60000);
                    }
                }
                else //=== Manifest is null ===//
                    bContinue = false;
            }
        }

        static IQueryable<IIngestManifest> ListIngestManifests(string manifestId = "")
        {
            CloudMediaContext context = GetContext();

            Console.WriteLine("\n===============================================");
            Console.WriteLine("===========[ BULK INGEST MANIFESTS ]===========");
            Console.WriteLine("===============================================\n");

            IQueryable<IIngestManifest> manifests = null;

            //=== If an Id is supplied, list the manifest with that Id. Otherwise, list all manifests ===//
            if (manifestId == "")
                manifests = context.IngestManifests;
            else
                manifests = context.IngestManifests.Where(m => m.Id == manifestId);

            foreach (IIngestManifest manifest in manifests)
            {
                Console.WriteLine("Manifest Name  : {0}", manifest.Name);
                Console.WriteLine("Manifest State : {0}", manifest.State.ToString());
                Console.WriteLine("Manifest Id    : {0}", manifest.Id);
                Console.WriteLine("Manifest Last Modified      : {0}", manifest.LastModified.ToLocalTime().ToString());
                Console.WriteLine("Manifest PendingFilesCount  : {0}", manifest.Statistics.PendingFilesCount);
                Console.WriteLine("Manifest FinishedFilesCount : {0}", manifest.Statistics.FinishedFilesCount);
                Console.WriteLine("Manifest BLOB URI : {0}\n", manifest.BlobStorageUriForUpload);

                foreach (IIngestManifestAsset manifestasset in manifest.IngestManifestAssets)
                {
                    Console.WriteLine("\tAsset Name    : {0}", manifestasset.Asset.Name);
                    Console.WriteLine("\tAsset ID      : {0}", manifestasset.Asset.Id);
                    Console.WriteLine("\tAsset Options : {0}", manifestasset.Asset.Options.ToString());
                    Console.WriteLine("\tAsset State   : {0}", manifestasset.Asset.State.ToString());
                    Console.WriteLine("\tAsset Files....");

                    foreach (IIngestManifestFile assetfile in manifestasset.IngestManifestFiles)
                    {
                        Console.WriteLine("\t\t{0}\n\t\tFile State : {1}\n", assetfile.Name, assetfile.State.ToString());
                    }
                    Console.WriteLine("");
                }
            }

            return manifests;
        }
    }
}

App.Config file example

ImportantImportant
Before running the sample, you need to add values for MediaServicesAccountName, MediaServicesAccountKey, MediaServicesStorageAccountName, and MediaServicesStorageAccountKey. These values can be found by going to the Azure Portal select your Azure Media Service account and click the Manage Keys icon. Do the same for the Azure Storage account.

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0" />
  </startup>
  <appSettings>
    <add key="MediaServicesAccountName" value="Enter Your Azure Media Services Account Name " />
    <add key="MediaServicesAccountKey" value="Enter Your Azure Media Services Account Key " />
    <add key="MediaServicesStorageAccountName" value=" Enter the Storage Account Name used with Azure Media Services " />
    <add key="MediaServicesStorageAccountKey" value=" Enter the Storage Account Key used with Azure Media Services " />
  </appSettings>
</configuration>

Example output from the example code.


===============================================
========[ CREATE BULK INGEST MANIFEST ]========
===============================================


===============================================
===========[ BULK INGEST MANIFESTS ]===========
===============================================

Manifest Name  : TestManifest
Manifest State : Activating
Manifest Id    : nb:mid:UUID:47e2efdf-8ec8-6143-a4ac-bf43efba35c8
Manifest Last Modified      : 2/25/2013 1:50:09 PM
Manifest PendingFilesCount  : 3
Manifest FinishedFilesCount : 0
Manifest BLOB URI : https://mediasvcm13w6rsvm521t.blob.core.windows.net/manifest
-402dcea0-411c-48f2-90fc-805c63a92a72

        Asset Name    : TestManifest_asset_2
        Asset ID      : nb:cid:UUID:0dbfcaf9-e53c-4399-99c2-ba207b1821d9
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                interview2.wmv
                File State : Pending

                interview1.wmv
                File State : Pending


        Asset Name    : TestManifest_asset_1
        Asset ID      : nb:cid:UUID:7a7188b5-d39d-42de-b3c7-f418174b14f3
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                BigBuckBunny.mp4
                File State : Pending



===============================================
===[ BULK INGEST MANIFEST MONITOR FILE COPY]===
===============================================


Waiting on all file uploads.
PendingFilesCount  : 3
FinishedFilesCount : 0
0% complete.

Upload for c:\Projects\Media Services\BulkIngestTest\supportFiles\multifile\inte
rview2.wmv completed.
Upload for c:\Projects\Media Services\BulkIngestTest\supportFiles\multifile\BigB
uckBunny.mp4 completed.
Upload for c:\Projects\Media Services\BulkIngestTest\supportFiles\multifile\inte
rview1.wmv completed.

Waiting on all file uploads.
PendingFilesCount  : 0
FinishedFilesCount : 3
100% complete.

Completed


===============================================
===========[ BULK INGEST MANIFESTS ]===========
===============================================

Manifest Name  : TestManifest
Manifest State : Inactive
Manifest Id    : nb:mid:UUID:47e2efdf-8ec8-6143-a4ac-bf43efba35c8
Manifest Last Modified      : 2/25/2013 1:50:48 PM
Manifest PendingFilesCount  : 0
Manifest FinishedFilesCount : 3
Manifest BLOB URI : https://mediasvcm13w6rsvm521t.blob.core.windows.net/manifest
-402dcea0-411c-48f2-90fc-805c63a92a72

        Asset Name    : TestManifest_asset_2
        Asset ID      : nb:cid:UUID:0dbfcaf9-e53c-4399-99c2-ba207b1821d9
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                interview2.wmv
                File State : Finished

                interview1.wmv
                File State : Finished


        Asset Name    : TestManifest_asset_1
        Asset ID      : nb:cid:UUID:7a7188b5-d39d-42de-b3c7-f418174b14f3
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                BigBuckBunny.mp4
                File State : Finished



===============================================
===========[ BULK INGEST MANIFESTS ]===========
===============================================

Manifest Name  : TestManifest
Manifest State : Inactive
Manifest Id    : nb:mid:UUID:c44efbcc-8f6a-f94a-aa21-2d0c1166ecb7
Manifest Last Modified      : 2/25/2013 1:47:47 PM
Manifest PendingFilesCount  : 0
Manifest FinishedFilesCount : 0
Manifest BLOB URI : https://mediasvcm13w6rsvm521t.blob.core.windows.net/manifest
-f001a80c-3c37-4a31-8ffc-b4bfe98b56be

        Asset Name    : TestManifest_asset_1
        Asset ID      : nb:cid:UUID:350ceed6-086a-413d-9bb9-a8a2f07a60f5
        Asset Options : None
        Asset State   : Initialized
        Asset Files....

Manifest Name  : TestManifest
Manifest State : Inactive
Manifest Id    : nb:mid:UUID:47e2efdf-8ec8-6143-a4ac-bf43efba35c8
Manifest Last Modified      : 2/25/2013 1:50:48 PM
Manifest PendingFilesCount  : 0
Manifest FinishedFilesCount : 3
Manifest BLOB URI : https://mediasvcm13w6rsvm521t.blob.core.windows.net/manifest
-402dcea0-411c-48f2-90fc-805c63a92a72

        Asset Name    : TestManifest_asset_2
        Asset ID      : nb:cid:UUID:0dbfcaf9-e53c-4399-99c2-ba207b1821d9
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                interview2.wmv
                File State : Finished

                interview1.wmv
                File State : Finished


        Asset Name    : TestManifest_asset_1
        Asset ID      : nb:cid:UUID:7a7188b5-d39d-42de-b3c7-f418174b14f3
        Asset Options : None
        Asset State   : Initialized
        Asset Files....
                BigBuckBunny.mp4
                File State : Finished



===============================================
=======[ DELETE BULK INGEST MANIFESTS ]========
===============================================


===============================================
=====[ DELETE BULK INGEST MANIFEST ASSETS ]====
===============================================

Deleting assets for manifest named : TestManifest...

Deleting asset nb:cid:UUID:350ceed6-086a-413d-9bb9-a8a2f07a60f5

Deleting Manifest...
        Name : TestManifest
        Manifest ID : nb:mid:UUID:c44efbcc-8f6a-f94a-aa21-2d0c1166ecb7...
Delete Complete.


===============================================
=====[ DELETE BULK INGEST MANIFEST ASSETS ]====
===============================================

Deleting assets for manifest named : TestManifest...

Deleting asset nb:cid:UUID:0dbfcaf9-e53c-4399-99c2-ba207b1821d9

Deleting asset nb:cid:UUID:7a7188b5-d39d-42de-b3c7-f418174b14f3

Deleting Manifest...
        Name : TestManifest
        Manifest ID : nb:mid:UUID:47e2efdf-8ec8-6143-a4ac-bf43efba35c8...
Delete Complete.

You are ready to proceed to the next topic: Static Packaging.

See Also


Build Date:

2014-07-11

Community Additions

ADD
Show:
© 2014 Microsoft