The Ins and Outs of Script Concatenation

Christophe Porteneuve | May 11, 2011

If you’ve run any sort of technical web page scoring tool lately, you know that serving numerous script files is regarded as quite the performance fail these days. Loading them asynchronously and using so-called asset hosts or a bona-fide CDN are useful tricks but only mitigate the issue: it avoids blocking and increases parallel download channels, but doesn’t cut down on the actual network requests.

On the other hand, stuffing all your script in one giant source file is a maintenance nightmare (having to wade through a huge source file, whatever your need, is no fun). Plus, your target platform may refuse to cache files that are too big. An iPhone 4 will not cache anything above 25KB, that’s fairly small! Before version 4, it topped up at a measly 15KB. This is another reason why using desktop-oriented JavaScript frameworks doesn’t make a lot of sense when targeting mobile browsers.

Efficient script management is all about that sweet spot where we can both manage our JavaScript codebase as a cleanly-structured source tree, and serve it with the right balance of file count and file size (before network compression, by the way).

This article attempts to shine a light on best practices for serving your scripts in an effective way. We will not, however, dive into the timeliness of serving your scripts (synchronously, asynchronously, from the head or at strategic places in the body, or on-demand). That’s another question entirely, the gist of the answer to it being “as late and async as you can, while honoring dependencies and not botching your DOM loaded events.”

Splitting your code into manageable chunks

A major benefit of using a toolchain to deliver properly-served chunks of JavaScript is that you can divide up your script codebase in as many files as you want. Just like with server-side languages, I like to use a directory tree to represent “packages,” and cut up my code until every file is focused enough that its entire code fits on a couple of pages, if not just one.

With sufficient granularity, it becomes easier to “learn” a codebase (when you’re a new contributor to the project just getting your bearings, for instance) as you only need to wrap your head around one reasonably-sized chunk of code at a time. Not only that, but this lets you name your folders and files in a topic-based way that makes it easier to locate code when you know your topic but now where the relevant code was tucked away.

As always, naming is entirely up to you. Pick something that makes sense to you and stick to it. Perhaps you can adopt well-adopted conventions here and there, to make things easier for others. I like to use the following conventions:

• I use folder names ui for UI-related stuff (widgets, Web Forms 2 emulation, animations, etc.), vendor for external libraries that I chose to bundle instead of serve through external URLs, then a folder for each complex topic (e.g. a complex registration process)
• Whenever I have a number of scripts for a given topic, I create both a folder and a .js file at the same level with the same root name. The .js file then “includes” (or requires, if you prefer to think of it that way) all the scripts on the lower level, in whatever order I deem best. This way, other scripts don’t need to know the details of how I cut up my topic-related codebase: they can only include/require my upper-level script.

Another good habit is to wrap all your scripts in a module pattern. For those of you not familiar still with this major JavaScript topic, here’s a fantastic read about JavaScript Module Patterns that covers it from basic usage to advanced variants that had me go “Wowza!” (Twitter’s Ben Cherry sure knows his stuff). This can lead to various hurdles as your previous scripts were perhaps bigger and needed to share some private state. Ben’s article covers how to do that too, so you’re out of excuses for not making small, manageable files.

Here’s an partial tree from one of my recent projects:

  |-- application.js
|-- customer
|   |-- alert_slots.js
|   |-- alerts
|   |   |-- countdown.js
|   |   |-- hiders.js
|   |   |-- popups.js
|   |   |-- social_media.js
|   |   `-- toggling.js
|   |-- alerts.js
...
|   |-- topics
|   |   |-- fetcher.js
|   |   |-- first_time_safeguard.js
|   |   `-- subscriptions.js
|   `-- topics.js
|-- customer.js
...
`-- vendor
    |-- prototype.js
    |-- raphael    |   |-- graphael-full.js
    |   |-- plugins
    |   |   `-- raphael.path.methods.js
    |   `-- raphael.js
    |-- raphael.js
    `-- s2.js
  

Notice how I have a customer.js at the same level as the customer folder, and a customer/alerts.js at the same level as customer/alerts (same goes for customer/topics or vendor/raphael). I think of it as extra encapsulation. Then again, suit your taste!

Depending on your app, you may want to bundle your scripts in just one file, or a couple files. This may be because your codebase addresses several well-separated situations (e.g. frontoffice and backoffice, or more generally public vs. private/authenticated areas), or because you need to keep individual files below a certain target size so they can be properly cached on the client side. Or both.

Sprockets and Jammit

In this article we’re going to use two separate tools for processing and bundling or scripts to obtain just the target JavaScript files we need (properly compacted, too!).

Sprockets is a great tool written by Sam Stephenson, creator of the Prototype JavaScript library and the lead JavaScripter behind all 37signals applications. It’s been around for almost 2 years and is very successfully used in production in numerous websites big and small. It’s agnostic and can be used in any context.

Jammit is a Rails-specific plugin by JavaScripter extraordinaire Jeremy Ashkenas (also behind the outstanding Backbone.js and Underscore.js libraries and mind-blowing CoffeeScript–the guy just never rests) that deals with both JavaScript and CSS, and also provides a number of fairly advanced optimization features.

Both are written in Ruby (therefore readily available as Rubygems), open-source and available for download and your forking pleasure on GitHub.

Using Sprockets

Sprockets’ current stable version, 1.0, is focused on JavaScript itself, although it has a notion of companion assets that it can copy over at specified locations when packaging. It can be used in just about any context as it comes with a command-line tool designed for automated packaging (for instance, in deployment hooks), a Rails plugin for stupid-easy on-demand packaging, and a CGI/FastCGI interface for on-demand use in all other webapp technologies.

You need Ruby on your machine to run it. If you’re on OS X or Linux, it’s already there. If you’re on Windows, you may need to run the super-simple Ruby Installer to get all set up. Then just run:

gem install sprockets

Sprocket’s upcoming version 2 covers a lot more though, with support for CoffeeScript, CSS, LESS and more. It also favors an in-webapp approach and provides a Rack middleware for plug-and-play use with most application servers these days. Although not officially released yet, it’s fairly stable and is used in several production apps already, including the acclaimed mobile version of Basecamp.

Requires

In Sprockets, you “import” a script into another one through requires. This actually behaves like an include, so the contents of the required script ends up where you require is at. To achieve this, you use a directive, which in Sprockets relies on special comment-based syntax: “//= require”.

Sprockets supports a notion of load path, much like the PATH variable you’d see in a shell or a C environment, or Java’s CLASSPATH variable. If you wrap your required script’s path with angular brackets (e.g. //= require <prototype>), the target file (in this case prototype.js) will be looked up in every directory listed in the load path. The other option is to wrap your path with regular double quotes (e.g. //= require "customer/topics"), that will look up for customer/topics.js relative to the current file’s directory. This is a common scheme in many languages.

Obviously, required files can themselves require other files, to any necessary depth. Concatenation will start with all the “root” files you passed it and walk down their require directives recursively, adding up to a single result script. For instance, comments aside, here’s the original ajax.js source script in version 1.7 of the Prototype JavaScript library:

//= require "ajax/ajax"
//= require "ajax/responders"
//= require "ajax/base"
//= require "ajax/request"
//= require "ajax/response"
//= require "ajax/updater"
//= require "ajax/periodical_updater"

This lets outer scripts just require ajax without having to know the innards of it, and lets us split Ajax functionality with the desired level of granularity.

Constants

Sprockets also lets you declare constants and reuse those anywhere in your source scripts. Common uses for this include version numbers, copyright information, website URLs and color codes. All you have to do is put a constants.yml file somewhere in your load path (usually at the root level) where you declare your constants, and then refer to those in your scripts with an ERb-like expression syntax (<%= MyConstant %>).

The constants.yml file uses, as its extension implies, the YAML syntax. Unless you have fairly advanced needs, this just means name-value pairs separated by colons. Prototype 1.7’s constants file is trivial:

  PROTOTYPE_VERSION: 1.7

Concatenating

The most basic way of processing your source tree for concatenation is through the gem-provided binary sprocketize. Running it with no arguments or with --help will detail possible command-line options:

  $ sprocketize
Usage: sprocketize [options] filename [filename ...]
    -C, --directory=DIRECTORY    Change to DIRECTORY before doing anything
    -I, --include-dir=DIRECTORY  Adds the directory to the Sprockets load path
    -a, --asset-root=DIRECTORY   Copy provided assets into DIRECTORY
    -h, --help                   Shows this help message
    -v, --version                Shows version
  

  

You just pass it a list of “root” source scripts to preprocess, and it will output the concatenated result on the standard output stream. If you want to specify a load path (the default is just the current directory), use the -I option for each component directory. Here are a couple example calls:

$ sprocketize application.js > sprockets.js

$ sprocketize common.js frontoffice.js > front.js
$ sprocketize common.js backoffice.js  > back.js

$ sprocketize -I vendor/plugins/*/javascripts -I app/javascripts application.js > public/sprockets.js

Notice that by default, Sprockets will expand load path components using shell globbing rules (processing wildcards such as ?, * and **).

Finally, know that concatenation itself does just that: preprocessing and concatenating your scripts. No compacting happens, so if you need that, you’ll have to slap your favorite JavaScript compressor on top of the result. Personally, I recommend UglifyJS or YUI:Compressor (also available wrapped in a Ruby gem).

Images and stylesheets: companion assets

Your script may require “companion assets” to work, especially if provides UI. This could be an image cropper, for instance, or a WYSIWYG editor of some sort. Your script is not self-sufficient then, and requiring it should also make its companion assets available. Sprockets caters to that need through the //= provide directive.

Any script using that directive to point to a directory of companion assets triggers asset deployment when Sprockets requires it. The assets get copied over, recursively, to an asset root directory you specified. With the sprocketize command, you would use the -a option to do that.

As an example, let’s say you have the following situation:

  |-- application.js
`-- vendor
    `-- cropper
        |-- cropper.js
            `-- assets
                |-- images
                |   `-- cropper_sprited.gif
                `-- stylesheets
                    `-- cropper.css
  

  

Now suppose your application.js has a //= require <vendor/cropper/cropper.js> directive, and cropper.js has a //= provide "assets" directive. Then you run the following command line:

$ sprocketize -a public application.js > public/sprockets.js

This will:

1. Concatenate all the relevant scripts and put the result in public/sprockets.js
2. Copy over the contents of vendor/cropper/assetsinside the public directory. So you’ll end up with public/images/cropper_sprited.gif and public/stylesheets/cropper.css.

Automatically serving fresh on any web server

Processing our source scripts and assets to obtain optimized files is something best done as an automated task upon deployment. However, you could want to have this done automagically for you when requiring the target script’s URL. This is especially true outside production mode. Sprockets comes with two ways to do this: a generic CGI-based solution, and an extra gem providing Rails-specific help.

The Sprockets gem contains a ext/nph-sprockets.cgi file, written in Ruby, that you can set up using a config/sprockets.yml file, then bind to whichever address you like (e.g. /sprockets.js). It can cache its result to disk, or be entirely in-memory and reprocess your source tree on every request (something you definitely don’t want to do in production). The CGI file starts with detailed instructions on how to configure it and set it up in your typical Apache environment.

Extra help within a Rails app

If you’re fortunate enough to use Rails for your web apps, there’s a companion gem called sprockets-rails that makes it a snap to bind your /sprockets.js URL to Sprockets behavior configured through a config/sprockets.yml file. The default configuration goes like this:

:asset_root: public
:load_path:
  - app/javascripts
  - vendor/sprockets/*/src
  - vendor/plugins/*/javascripts
  - vendor/plugins/*/app/javascripts
:source_files:
  - app/javascripts/application.js
  - app/javascripts/**/*.js

This defines a few conventions for your load path, including a Sprockets-specific plugin type in vendor/sprockets. It also states that your Sprockets-processed scripts are not available directly in the usual public/javascripts directory, but stay in the private app/javascripts path, and that the root file is application.js. I usually kill the last line, as I prefer only to include files explicitly, and in the order of my choice, through require directives.

Version 1 vs. version 2

Version 1 has been stable for a long time, and all the current work is going into upcoming version 2, which works in a different way. It’s provided as a Rack middleware, making it stupid-simple to plug into any Rack-compatible web server. (There are adapters for Mongrel, WEBrick, FCGI/CGI/SCGI, Thin, Glassfish, Passenger, Unicorn, and more!)

It also expands significantly upon version 1, as it handles both JavaScript (regular or CoffeeScript) and CSS (regular, LESS or SCSS), and interacts with compressor for both file types. A final good point is that it lets you work with multiple target files instead of just one, as version 1’s CGI/Rails helpers did. Asset handling doesn’t rely on copy anymore: it’s able to serve assets directly from your Sprockets load path. Documentation isn’t there at the time of this writing, but it’s solid already, so if you want to play with it, head over to GitHub.

Using Jammit

Jammit is another strong contender in the script concatenation space. Although Rails-specific, it’s a great enough solution that I felt I should cover it here. It has a lot of features and options that we won’t have enough room to explore, but the basics should get you interested enough to browse the docs for more.

Installing Jammit is as simple as gem install jammit. Then don’t forget to add it to your Rails environment (either through your config/environment.rb or Gemfile, depending on which Rails version you run). Also, if you’re not running Rails 3, you’ll need to bind Jammit to a route (which defaults to /assets) by adding a call in your config/routes.rb file:

ActionController::Routing::Routes.draw do |map|
  ...
  Jammit::Routes.draw(map)
  ...
end

Jammit will auto-compress your concatenated assets if it can find a suitable compressor in your Ruby environment. Again, I recommend the ruby-yui-compressor gem.

Configuring your assets

You configure Jammit using the config/assets.yml file. Here’s a regular-complexity example from the docs:

javascripts:
  workspace:
    - public/javascripts/vendor/jquery.js
    - public/javascripts/lib/*.js
    - public/javascripts/views/**/*.js
    - app/views/workspace/*.jst

stylesheets:
  common:
    - public/stylesheets/reset.css
    - public/stylesheets/widgets/*.css
  workspace:
    - public/stylesheets/pages/workspace.css
  empty:
    - public/stylesheets/pages/empty.css

There are a few things to notice here:

• Scripts and stylesheets can be grouped into “targets,” each of which can be served individually. This make splitting (e.g. homepage, frontoffice and backoffice) a snap.
• Globbing patterns (* and **) are allowed here too.
• Both JavaScript and stylesheets are handled.
• A built-in templating mechanism is supported, defaulting to a variant of John Resig’s JavaScript Micro-Templating. You can switch it to equivalent features in Underscore.js, Prototype or Mustache.js if you wish to, or disable it entirely.

Serving your stuff with Jammit

Jammit provides two additional helper methods: include_javascripts and include_stylesheets, that can take one or more targets at once:

<%= include_stylesheets :common, :workspace, :media => 'all' %>
<%= include_javascripts :workspace %>

By default, Jammit will not package source files into single-file targets in development mode, to ease your debugging. You can change that by setting the package_assets option to always. You can still get to non-packaged individual files on specific requests by putting a debug_assets=true parameter in the request, unless you explicitly disabled that feature in your configuration.

Nifty features

Jammit has a few more options and features. Here’s a full-on configuration example.

It can be invoked from Ruby code so you can package stuff in advance (as a deployment task for instance), can compress your JavaScript and CSS using either YUI::Compressor or Closure, can pre-gzip your target files, and even embed your images and fonts in your CSS files using either Data URIs or MHTML, depending on the browser.

Be careful though: while this does considerably reduce your number of network requests, this can quickly push your CSS files above the cachability size limit of mobile devices. So if that’s a concern, tweak your settings until you find the right balance between file count and file size.

Finally, as we mentioned earlier, it provides a level of integration with four types of JavaScript templating (a built-in variant of Micro-Templating, Underscore.js, Prototype or Mustache.js), as described in this section of the docs. You can tell how this is influenced by the needs of the Backbone.js client-side MVC framework, by the same authors…

Conclusion

In this area of client-side web page optimization, you can have your cake and eat it too: structure your original JavaScript (and CSS) to any desired degree of modularity/granularity in your source tree, whilst serving it as a well-tuned smaller set of cachable, compacted resources to the client side. And you don't even need to sacrifice the comfort and productivity of higher-level syntax such as CoffeeScript or LESS!

There’s just no reason to wade through huge scripts and stylesheets anymore.

About the Author

At age 33, Christophe has been doing pro web development for about 15 years. A JavaScript enthusiast, he is a member of Prototype Core and the author of several web development books, including Prototype and script.aculo.us: you never knew JavaScript could do this! and Pragmatic Guide to JavaScript at the Pragmatic Bookshelf.  Christophe speaks at various web conferences such as The Ajax Experience, JSConf.eu and Paris Web.  He also recently expounded on the current state of JS on JSConfLive.

His current job is CTO of Ciblo, a Rails-centric web agency.  He lives in Paris with his wife Élodie.

Find Christophe on:

• Twitter - @porteneuve