July 2014

Volume 29 Number 7

Azure Web Sites : Building a Node.js and MongoDB Web Service

Tejaswi Redkar

The cloud is neutral to all languages. In the cloud, it shouldn’t matter whether the apps you run are Node.js, ASP.NET, Java or PHP, because the cloud provides a ready-made infrastructure for running them. The religion of the cloud is agility, and agility allows you to implement new ideas quickly and sunset the ones that don’t work in the marketplace. In recent years, Node.js has gained popularity among a new class of developers who favor the asynchronous programming paradigm. Node.js is a JavaScript runtime engine for building client and server applications. It includes an asynchronous I/O framework that helps it handle thousands of concurrent connections on a single thread with minimal CPU or memory overhead. It’s a popular myth that Node.js is a Web development framework; in fact, it’s a single-threaded runtime engine for running JavaScript code. It has a module-based architecture that allows anyone to build and publish a module to handle specific tasks, such as Web development or accessing a MongoDB database. For example, the Express template engine is a Web application framework you can download as a module in Node.js. The core Node.js engine is built in C++ on top of the Google V8 JavaScript engine, which was designed for the Chrome browser. Node.js is supported in Microsoft Azure Web Sites, and Microsoft also provides development tools and a native SDK for programming with Azure APIs. There’s a dedicated developer center for Node.js on the Azure Portal at bit.ly/1iupElQ.

In this article, I’ll show you how to develop a RESTful Web service in Node.js that accesses a MongoDB database in the cloud, and how to deploy it to Azure Web Sites.

Node.js is based on an asynchronous development model, which means every method call you make requires a callback to receive the response. Though .NET developers traditionally prefer synchronous (request-response) method calls, asynchronous capabilities have always existed in the Microsoft .NET Framework. With the inclusion of the new async-await programming model in the .NET Framework, asynchronous applications have become the norm across Web and mobile applications. In asynchronous programming, the calling function subscribes to the callback event and provides a delegate to process the response. The callback function is called when the processing is complete. It’s like an e-mail compared to a phone call.

The following shows a simple Node.js Web server that returns a string when called:

var http = require("http");
http.createServer(function(request, response) {
  response.writeHead(200, {"Content-Type":
  "text/plain"});
  response.write("Hello MSDN");
  response.end();
}).listen(8080);

Note that the response function is called when an HTTP request event is fired. The require function is used for loading modules, similar to loading namespaces from assemblies in the .NET Framework.

There have been a lot of discussions and religious arguments on ASP.NET versus Node.js, but I’m not going to touch that topic in this article. My approach has always been to use the best tool for the job, and with what you’re good at do your best.

Node.js Request Processing

As Figure 1 shows, the Node.js engine starts a single thread for handling concurrent client connections. As the single thread is already initialized, there’s no initialization overhead required for processing any rise in requests because the thread quickly delegates the request asynchronously to a worker thread for processing.

Figure 1 Node.js Request-Response Orchestration

If the HTTP request incudes a long-running or I/O-intensive task, such as database access or a Web service call, it’s executed asynchronously in non-blocking worker threads. Once the long-running task is complete, the worker threads return the results as a callback to the main thread. The main thread then returns the result back to the client. An impor­tant concept to understand here is that the single receiving thread is always available to receive requests and doesn’t remain busy with processing because processing is delegated to the worker threads.

Node.js Core Components

Node.js consists of two core components:

Core/Kernel: The kernel of Node.js is written in C++ on top of the Google V8 JavaScript engine. The core itself is single-threaded and is capable of load balancing among CPUs. Node.js is open source and you can get the source code from github.com/joyent/node.

Modules: Modules are similar to NuGet packages in the .NET Framework. The Node.js Package Manager (NPM) is the tool for managing Node.js packages in your development environment. Modules spawn new processes or threads depending on the I/O intensity of the task. Among the popular modules are HTTP, MongoDB, Express (a Web template framework) and Socket.IO. For a list of popular modules, please visit nodejsmodules.org.

Installing and Running Node.js Locally

Before running any Node.js Web site in the cloud, I recommend trying it out locally to get comfortable with the platform. You can install and run Node.js on a Windows platform in just three steps:

1. Download and install Node.js: The Node.js Windows installer can be downloaded and installed from nodejs.org/#download. The installer installs the Node.js runtime and NPM.
2. Create a simple server: To create an HTTP Server in Node.js, open your favorite text editor, copy the code from Figure 1, and save the file as webserver.js. The code shown earlier creates a minimalist Web server that listens on port 8080 and responds with the same string for every request.
3. Run the HTTP server: Finally, to run the server, open command prompt, navigate to the folder where you saved the webserver.js file and type the following:
               >“C:\Program Files\nodejs\node.exe” webserver.js
4. This command starts the Web server, and you can test it by navigating your browser to https://localhost:8080. This simple server should give you enough confidence to try out Node.js as an option for developing Web sites.

Running Node.js on Azure Web Sites

When I first used Node.js, I decided to avoid the command line as much as possible. Having lived my life in Visual Studio, I really value the productivity you can achieve using an IDE. Fortunately, Microsoft has invested in WebMatrix, a powerful tool for devel­oping Node.js applications on Windows. The Visual Studio team has also released the Node.js tools for Visual Studio (nodejstools.codeplex.com). For the rest of this article, I’ll use WebMatrix as my primary development tool. From WebMatrix, you can install NPM packages, publish Web sites to Azure, and also run them locally. WebMatrix also installs IISNode for IIS Express, which allows you to host the Node.js application on IIS. You can get more details about IISNode at github.com/tjanczuk/iisnode.

Before building a complete RESTful Web service, I’ll show you how to publish a simple Web site to Azure from WebMatrix.

Create a New Azure Web Site You can create a new Web site in Azure from the Azure Portal, as illustrated in Figure 2.

Figure 2 Create a New Azure Web Site

The Portal will create a new site with a unique name in the region you specify. The region is important for co-locating your Web site, databases and other services in the same datacenter. Data that leaves a datacenter is charged.

Create an Express Web Site Express is a Web application framework for Node.js. It follows the Model-View-Controller (MVC) pattern and, therefore, allows you to establish routes for building Node.js MVC Web sites as well as RESTful Web services. You can download Express from expressjs.com.

If you love .NET development in Visual Studio, I recommend mastering WebMatrix for developing Node.js applications. WebMatrix 3 includes a useful, prebuilt Express template. Open it and click on New | Template Gallery. Then, under the Node.js category, select the Express Site template, as shown in Figure 3.

Specify a site name and click Next to install IISNode and the Express Site template.

Figure 3 Express Site Template

The Node.js server I built earlier won’t run as is on Azure Web Sites because the Azure Web Sites infrastructure relies on IIS for running Web sites. Therefore, to run a Node.js Web site, I need the integration between IIS and Node.js that IISNode provides.

Figure 4 illustrates the file structure created by the Express template, and the source code for server.js.

Figure 4 Express File Structure

Note that the express and routes modules get imported automatically. I’ll use these modules to build the REST services.

Running the Web Site Locally Following my earlier recommendation, click the Run button on WebMatrix to test the Web site on your local machine. If the installation was successful, you should see the Express homepage.

Express template also installs the Jade template engine. Jade is an HTML template engine and is used for building the views generated from the Express framework. Jade is to Express what Razor is to ASP.NET MVC. Therefore, the index.html contents are rendered from /views/index.jade and /routes/index.js. These routes are set up on lines 16, 17 and 30 of server.js, as shown in Figure 4. For more information on the Jade template engine, visit jade-lang.com.

Publishing the Web site to Azure Now that you’ve built a Node.js Web site in Express locally, let’s publish it to Azure. Click on the Publish button in WebMatrix to start the process. Import the publish profile for your Web site and follow the publishing wizard as shown in Figure 5.

Figure 5 Publish to Microsoft Azure

If everything goes well, WebMatrix will load your Web site in Internet Explorer.

The URL in the browser should be the Azure Web Site to which you published the Web site. This is a good demonstration of how easy it is to develop and deploy Node.js Web sites and Web services to Azure. Now, I’ll take a moment to look at MongoDB. I’ll come back to Node.js later to build Web services.

MongoDB Overview

MongoDB is an open source, scalable, high-performance, document-oriented database that includes most of the infrastructure capabilities of relational databases, such as replication, sharding, indexing and failover. MongoDB also offers auto-sharding and has built-in Map-Reduce processing capabilities, which aren’t offered by most relational databases today. Relational databases were designed in an era where storage systems were expensive. Storing data in a relational format allowed developers to optimize storage space without compromising quick data retrieval. In today’s world, storage is cheap when compared to compute costs. In the cloud, storage prices continue to decline. As a result, saving and retrieving data in relational format is expensive compared with storing it in the cloud. The demand for continuous data delivery is on the rise, with more applications expecting real-time data from your data storage. MongoDB allows you to save objects and documents without breaking them down into relational components. This reduces the processing load on the application, as well as the database.

MongoDB isn’t recommended for applications that require deep object relationships because it isn’t designed to maintain and retrieve object and data relationship linking, as relational databases are. If your application requires deep relations and SQL for retrieval, then use a relational database. If your application requires fast object storage and retrieval, use MongoDB. As I suggested earlier, use the best tool for the job.

Provisioning MongoDB in Azure

MongoDB is available as an add-on in the Azure Store and you can install a sandbox version for free. Log in to your Azure Portal and install MongoDB (MongoLab) from the New | Store | Add-on menu, as shown in Figure 6.

Figure 6 MongoDB Add-on from MongoLab

On the Personalize Add-on page, select the Sandbox version, and be sure to install the MongoDB instance in the same region as your Web site, as shown in Figure 7.

Figure 7 Configuring MongoDB

Once the installation is complete, navigate to the add-on page and save the connection string for the installed database, as shown in Figure 8. You’ll use the connection string to connect from your Node.js Web service.

Figure 8 Connection String for the Installed Database

MongoDB is now running in Azure. MongoLab (by ObjectLabs Corp.) has its own dedicated Management Portal you can view by clicking on the Manage Your Add-on link.

You now have a MongoDB database running in the cloud, and you’ve created an empty Node.js template. To complete the application, you need to populate the MongoDB database, and create the Web services that retrieve data from this database. This is standard software development in the cloud.

Installing Node.js Prerequisites for MongoDB Connectivity

JSON is a first-class citizen in MongoDB and, therefore, complements Node.js. Because the Web service will connect to MongoDB, I need to install the latest Node.js driver for MongoDB from the NPM Gallery (see Figure 9).

Figure 9 Node.js Driver for MongoDB

The MongoDB connection information gets stored in a configuration file. The nconf module allows you to read configuration information from files (see Figure 10).

Figure 10 The nconf Module

Once the mongodb and nconf modules are installed, you’ve completed all the prerequisites for setting up a Node.js and MongoDB development environment.

Creating the REST Web Service Signatures

To create a REST Web service, you first define dependencies and routes in server.js:

var express = require('express')
  , routes = require('./routes')
  , user = require('./routes/user')
  , http = require('http')
  , path = require('path'),
  pkgs=require('./routes/pkgs');

Next, you define the REST functions in the /routes/pkgs.js file because, in the preceding code, you delegate the function handling to the pkgs module. In the routes folder, create a file named pkgs.js that defines the Web service operations, as shown in Figure 11.

Figure 11 REST Operation Signatures

exports.findAll = function(req, res) {
  res.send([{name:'app1'}, {name:'app2'}, {name:'app3'}]);
};
exports.findById = function(req, res) {
  res.send({id:req.params.id, 
  name: "DisplayName", description: "description"});
};
exports.addPkg = function (req, res) {
  res.send("Success");
};
exports.updatePkg = function (req, res) {
  res.send("Success");
};

To test the basic functioning of the operations, just create function skeletons returning static data. You can add the database access code later. In server.js, define the route methods that correspond to the module methods you just created:

app.get('/pkgs', pkgs.findAll);
app.get('/pkgs/:id', pkgs.findById);
app.post('/pkgs', pkgs.addPkg);
app.put('/pkgs/:id', pkgs.updatePkg);

If you’re familiar with the ASP.NET Web API, you’ll quickly understand these routes and their mapping to the appropriate implementation class. With the route declaration and mapping, any request to /pkgs will be routed to one of the functions in pkgs.js. With the Web service operation signatures and definitions in place, you can test if the routes are working properly by running the Web site locally, as well as in Azure. The URLs to test are: https://[Web site URL]/pkgs, https://[Web site URL]/pkgs/remote, and https://[Web site Url]/pkgs/1. If these URLs return the expected values, the routes are working fine and you can proceed with the MongoDB integration.

Implementing REST Web Service Operations

Now you need to implement the Web service operations to connect to MongoDB and retrieve the data. Four steps will accomplish this:

1. Creating a config.json file and storing the MongoDB connection string in it.
2. Initializing the MongoDB client.
3. Populating MongoDB with sample data. In this step, the function makes an HTTP GET call to https://storage.appsforazure.com/appsforazureobjectstest/servicepackages.json and stores the retrieved data into the MongoDB instance.
4. Implementing the data access functions for retrieving and storing data in MongoDB.

Creating the config.json File The config.json file, which is similar to a .NET app.config but in JSON format, stores your configuration information. Add the following name-value pair to the new file:

{
  "MONGOLAB_URI" :
    "mongodb://nodeapps:xxxxxxxx.f.migd9GGB9Ck3M17jlkVCUVI-@ds027758.
    mongolab.com:27758/nodeapps"
}

The value is the MongoDB connection string you saved earlier from the Azure Portal.

In pkgs.js, you have to import the nconf module to read the configuration values:

var nconf = require('nconf');
nconf.env().file({ file: 'config.json' });
var connectionString = nconf.get("MONGOLAB_URI");

The connectionString variable can then be used by the functions to connect to the MongoDB database. Note that connectionString is declared as a global variable in pkgs.js so that all the functions can access it.

Initializing the MongoDB client: To initialize a connection to the MongoDB database, you have to import the mongodb module and call the connect method, as shown in Figure 12.

Figure 12 Connect to MongoDB Database

var mongo = require('mongodb');
var MongoClient = mongo.MongoClient
MongoClient.connect(connectionString, function (err, db) {
  if (err) throw err;
  if (!err) {
    console.log("Connected to 'pkgsdb' database");
    db.collection('pkgs', { strict: true }, 
      function (err, collection) {
      if (err) {
        console.log(
          "The 'pkgsdb' collection doesn't exist.
          Creating it with sample data...");
        populateDB(db);
      }
    });
  }
})

When the connection is successful, the db variable in Figure 12 contains the database connection object. The db.collection function tries to connect to a collection named pkgs. If pkgs doesn’t exist, a new one is created by calling the populateDB function. In a real-world application, you wouldn’t need to do this because the collection would actually be created before running the application. (A collection in MongoDB is a grouping of related documents, analogous to a database table in a relational database system.)

Populating MongoDB with Sample Data For this article, I used sample data available as a collection of software packages from dynamicdeploy.com. In the populateDB function, I load this collection in JSON format from https://storage.appsforazure.com/appsforazureobjectstest/servicepackages.json, as shown in Figure 13.

Figure 13 Populating the Database

var populateDB = function (db) {
  var body = "";
  var url =
    "https://storage.appsforazure.com/appsforazureobjectstest/servicepackages.json";
  http.get(url, function (res2) {
    res2.on('data', function (chunk) {
      body += chunk;
    });
    res2.on("end", function () {
    var pkgs = JSON.parse(body)
    db.collection('pkgs', function (err, collection) {
      collection.insert(pkgs, { safe: true }, 
      function (err, result) { });
    });
    });
      res2.on("error", function (error) {
        // You can log here further
    })
  });

Note that JSON is a first-class citizen in Node.js, so you don’t have to import any module to parse the JSON objects. The collection.insert function inserts the entire JSON document (var pkgs) into the MongoDB collection. MongoDB determines the schema of the objects at run time and makes intelligent decisions about storing them. In a relational database, you’d have to define the schema of the table before storing any data in it. A collection gives you the flexibility to modify the schema at run time just by changing the properties of the JSON objects. If the object properties change, MongoDB automatically applies those changes to the schema before storing the object. This is useful for building applications, such as social and the Internet of Things feeds, which dynamically change with changes in data types.

Implementing the Data Access Functions Finally, you need to implement the data access functions for the HTTP GET, POST and PUT functions declared earlier. The HTTP GET function is mapped to the findById and findAll functions in the Web service, as shown in Figure 14.

Figure 14 The Find Functions

exports.findById = function (req, res) {
  var id = req.params.id;
  console.log('Retrieving pkg: ' + id);
  MongoClient.connect(connectionString, function (err, db) {
    if (err) throw err;
    if (!err) {
        db.collection('pkgs', function (err, collection) {
          collection.findOne({ '_id': new BSON.ObjectID(id) },
            function (err, item) {
            res.send(item);
            });
        });
      }
    })
  };
exports.findAll = function(req, res) {
  MongoClient.connect(connectionString, function(err, db) {
  if(err) throw err;
  if(!err) {
    db.collection('pkgs', function(err, collection) {
      collection.find().toArray(function(err, items) {
        res.send(items);
      });
    });
    }
  })
};

The findById function retrieves an object by its Id, whereas the findAll function retrieves all the objects. Functions collection.find­One and collection.find retrieve the results of these operations from MongoDB, respectively. Similarly, Figure 15 shows the addPkg method, which is mapped to an HTTP POST, and the updatePkg method, which is mapped to an HTTP PUT.

Figure 15 Add and Update Functions

exports.addPkg = function (req, res) {
  var pkg = req.body;
  console.log('Adding pkgs: ' + JSON.stringify(pkg));
  MongoClient.connect(connectionString, function (err, db) {
    if (err) throw err;
    if (!err) {
      db.collection('pkgs', function (err, collection) {
        collection.insert(pkg, { safe: true }, function (err, result) {
          if (err) {
            res.send({ 'error': 'An error has occurred' });
          } else {
            console.log('Success: ' + JSON.stringify(result[0]));
            res.send(result[0]);
          }
        });
      });
    }
  })
};
exports.updatePkg = function (req, res) {
  var id = req.params.id;
  var pkg = req.body;
  console.log('Updating pkgs: ' + id);
  console.log(JSON.stringify(pkg));
  MongoClient.connect(connectionString, function (err, db) {
    if (err) throw err;
    if (!err) {
      db.collection('pkgs', function (err, collection) {
        collection.update({ '_id': new BSON.ObjectID(id) },
          pkg, { safe: true }, function (err, result) {
          if (err) {
            console.log('Error updating pkg: ' + err);
            res.send({ 'error': 'An error has occurred' });
          } else {
            console.log('' + result + ' document(s) updated');
            res.send(pkg);
          }
        });
      });
    }
  })
};

If you’ve followed the article so far, you should be able to implement a delete function on your own. I’ll leave that as an exercise for you. Save and run the Web site locally first and then publish it to Azure Web Sites to test each REST function. The complete source code for the REST Web service and the Web site is available at github.com/dynamicdeploy/appsforazure.

Once you have the end-to-end application running, you can scale out your MongoDB infrastructure (or service) depending on the user load. As this article demonstrates, you don’t need any infrastructure to deploy large-scale Node.js and MongoDB Web sites on Azure. The skeleton I provided in the accompanying source code will help you get started building Web services and Web sites in Node.js on Azure Web Sites.

Wrapping Up

The purpose of this article was to get you comfortable building Web services in Node.js on Azure Web Sites. Node.js and MongoDB complement each other and because both are available as Platform as a Service (PaaS), there’s no up-front infrastructure cost for building a complete Web site and scaling it out dynamically. As a developer, you can focus on building and consuming the Web services right from the cloud into your mobile apps. Therefore, I believe Azure Web Sites is PaaS done right.

Tejaswi Redkar is an author and a software developer. He currently works for Microsoft as a director of application platform strategy and communities. His book, “Windows Azure Web Sites: Building Web Apps at a Rapid Pace” (Dynamic Deploy LLC, 2013), is the most-comprehensive and best-selling on the topic. Redkar is also the creator of appsforazure Windows Store App and dynamicdeploy.com, where he experiences first-hand running production apps in the cloud. You can reach out to him at tejaswi_redkar@hotmail.com and follow him on Twitter at twitter.com/tejaswiredkar.

Thanks to the following technical experts for reviewing this article: Microsoft Azure Product Management Team