The Ins and Outs of Script Concatenation
Christophe Porteneuve | May 11, 2011
If you’ve run any sort of technical web page scoring tool lately, you know that serving numerous script files is regarded as quite the performance fail these days. Loading them asynchronously and using so-called asset hosts or a bona-fide CDN are useful tricks but only mitigate the issue: it avoids blocking and increases parallel download channels, but doesn’t cut down on the actual network requests.
This article attempts to shine a light on best practices for serving your scripts in an effective way. We will not, however, dive into the timeliness of serving your scripts (synchronously, asynchronously, from the
head or at strategic places in the
body, or on-demand). That’s another question entirely, the gist of the answer to it being “as late and async as you can, while honoring dependencies and not botching your DOM loaded events.”
Splitting your code into manageable chunks
With sufficient granularity, it becomes easier to “learn” a codebase (when you’re a new contributor to the project just getting your bearings, for instance) as you only need to wrap your head around one reasonably-sized chunk of code at a time. Not only that, but this lets you name your folders and files in a topic-based way that makes it easier to locate code when you know your topic but now where the relevant code was tucked away.
As always, naming is entirely up to you. Pick something that makes sense to you and stick to it. Perhaps you can adopt well-adopted conventions here and there, to make things easier for others. I like to use the following conventions:
- I use folder names
uifor UI-related stuff (widgets, Web Forms 2 emulation, animations, etc.),
vendorfor external libraries that I chose to bundle instead of serve through external URLs, then a folder for each complex topic (e.g. a complex registration process)
- Whenever I have a number of scripts for a given topic, I create both a folder and a
.jsfile at the same level with the same root name. The
.jsfile then “includes” (or requires, if you prefer to think of it that way) all the scripts on the lower level, in whatever order I deem best. This way, other scripts don’t need to know the details of how I cut up my topic-related codebase: they can only include/require my upper-level script.
Here’s an partial tree from one of my recent projects:
| |-- alert_slots.js
| |-- alerts
| | |-- countdown.js
| | |-- hiders.js
| | |-- popups.js
| | |-- social_media.js
| | `-- toggling.js
| |-- alerts.js
| |-- topics
| | |-- fetcher.js
| | |-- first_time_safeguard.js
| | `-- subscriptions.js
| `-- topics.js
|-- raphael | |-- graphael-full.js
| |-- plugins
| | `-- raphael.path.methods.js
| `-- raphael.js
Notice how I have a
customer.js at the same level as the
customer folder, and a
customer/alerts.js at the same level as
customer/alerts (same goes for
vendor/raphael). I think of it as extra encapsulation. Then again, suit your taste!
Depending on your app, you may want to bundle your scripts in just one file, or a couple files. This may be because your codebase addresses several well-separated situations (e.g. frontoffice and backoffice, or more generally public vs. private/authenticated areas), or because you need to keep individual files below a certain target size so they can be properly cached on the client side. Or both.
Sprockets and Jammit
Both are written in Ruby (therefore readily available as Rubygems), open-source and available for download and your forking pleasure on GitHub.
You need Ruby on your machine to run it. If you’re on OS X or Linux, it’s already there. If you’re on Windows, you may need to run the super-simple Ruby Installer to get all set up. Then just run:
Sprocket’s upcoming version 2 covers a lot more though, with support for CoffeeScript, CSS, LESS and more. It also favors an in-webapp approach and provides a Rack middleware for plug-and-play use with most application servers these days. Although not officially released yet, it’s fairly stable and is used in several production apps already, including the acclaimed mobile version of Basecamp.
In Sprockets, you “import” a script into another one through requires. This actually behaves like an include, so the contents of the required script ends up where you require is at. To achieve this, you use a directive, which in Sprockets relies on special comment-based syntax: “
Sprockets supports a notion of load path, much like the
PATH variable you’d see in a shell or a C environment, or Java’s
CLASSPATH variable. If you wrap your required script’s path with angular brackets (e.g.
//= require <prototype>), the target file (in this case
prototype.js) will be looked up in every directory listed in the load path. The other option is to wrap your path with regular double quotes (e.g.
//= require "customer/topics"), that will look up for
customer/topics.js relative to the current file’s directory. This is a common scheme in many languages.
Obviously, required files can themselves require other files, to any necessary depth. Concatenation will start with all the “root” files you passed it and walk down their
require directives recursively, adding up to a single result script. For instance, comments aside, here’s the original
//= require "ajax/ajax" //= require "ajax/responders" //= require "ajax/base" //= require "ajax/request" //= require "ajax/response" //= require "ajax/updater" //= require "ajax/periodical_updater"
This lets outer scripts just require
ajax without having to know the innards of it, and lets us split Ajax functionality with the desired level of granularity.
Sprockets also lets you declare constants and reuse those anywhere in your source scripts. Common uses for this include version numbers, copyright information, website URLs and color codes. All you have to do is put a
constants.yml file somewhere in your load path (usually at the root level) where you declare your constants, and then refer to those in your scripts with an ERb-like expression syntax (
<%= MyConstant %>).
constants.yml file uses, as its extension implies, the YAML syntax. Unless you have fairly advanced needs, this just means name-value pairs separated by colons. Prototype 1.7’s constants file is trivial:
The most basic way of processing your source tree for concatenation is through the gem-provided binary
sprocketize. Running it with no arguments or with
--help will detail possible command-line options:
Usage: sprocketize [options] filename [filename ...]
-C, --directory=DIRECTORY Change to DIRECTORY before doing anything
-I, --include-dir=DIRECTORY Adds the directory to the Sprockets load path
-a, --asset-root=DIRECTORY Copy provided assets into DIRECTORY
-h, --help Shows this help message
-v, --version Shows version
You just pass it a list of “root” source scripts to preprocess, and it will output the concatenated result on the standard output stream. If you want to specify a load path (the default is just the current directory), use the
-I option for each component directory. Here are a couple example calls:
Notice that by default, Sprockets will expand load path components using shell globbing rules (processing wildcards such as
Images and stylesheets: companion assets
Your script may require “companion assets” to work, especially if provides UI. This could be an image cropper, for instance, or a WYSIWYG editor of some sort. Your script is not self-sufficient then, and requiring it should also make its companion assets available. Sprockets caters to that need through the
//= provide directive.
Any script using that directive to point to a directory of companion assets triggers asset deployment when Sprockets requires it. The assets get copied over, recursively, to an asset root directory you specified. With the
sprocketize command, you would use the
-a option to do that.
As an example, let’s say you have the following situation:
| `-- cropper_sprited.gif
Now suppose your
application.js has a
//= require <vendor/cropper/cropper.js> directive, and
cropper.js has a
//= provide "assets" directive. Then you run the following command line:
- Concatenate all the relevant scripts and put the result in
- Copy over the contents of
publicdirectory. So you’ll end up with
Automatically serving fresh on any web server
Processing our source scripts and assets to obtain optimized files is something best done as an automated task upon deployment. However, you could want to have this done automagically for you when requiring the target script’s URL. This is especially true outside production mode. Sprockets comes with two ways to do this: a generic CGI-based solution, and an extra gem providing Rails-specific help.
The Sprockets gem contains a
ext/nph-sprockets.cgi file, written in Ruby, that you can set up using a
config/sprockets.yml file, then bind to whichever address you like (e.g.
/sprockets.js). It can cache its result to disk, or be entirely in-memory and reprocess your source tree on every request (something you definitely don’t want to do in production). The CGI file starts with detailed instructions on how to configure it and set it up in your typical Apache environment.
Extra help within a Rails app
If you’re fortunate enough to use Rails for your web apps, there’s a companion gem called
sprockets-rails that makes it a snap to bind your
/sprockets.js URL to Sprockets behavior configured through a
config/sprockets.yml file. The default configuration goes like this:
This defines a few conventions for your load path, including a Sprockets-specific plugin type in
vendor/sprockets. It also states that your Sprockets-processed scripts are not available directly in the usual
application.js. I usually kill the last line, as I prefer only to include files explicitly, and in the order of my choice, through
Version 1 vs. version 2
Version 1 has been stable for a long time, and all the current work is going into upcoming version 2, which works in a different way. It’s provided as a Rack middleware, making it stupid-simple to plug into any Rack-compatible web server. (There are adapters for Mongrel, WEBrick, FCGI/CGI/SCGI, Thin, Glassfish, Passenger, Unicorn, and more!)
Jammit is another strong contender in the script concatenation space. Although Rails-specific, it’s a great enough solution that I felt I should cover it here. It has a lot of features and options that we won’t have enough room to explore, but the basics should get you interested enough to browse the docs for more.
Installing Jammit is as simple as
gem install jammit. Then don’t forget to add it to your Rails environment (either through your
Gemfile, depending on which Rails version you run). Also, if you’re not running Rails 3, you’ll need to bind Jammit to a route (which defaults to
/assets) by adding a call in your
ActionController::Routing::Routes.draw do |map| ... Jammit::Routes.draw(map) ... end
Jammit will auto-compress your concatenated assets if it can find a suitable compressor in your Ruby environment. Again, I recommend the
Configuring your assets
You configure Jammit using the
config/assets.yml file. Here’s a regular-complexity example from the docs:
There are a few things to notice here:
- Scripts and stylesheets can be grouped into “targets,” each of which can be served individually. This make splitting (e.g. homepage, frontoffice and backoffice) a snap.
- Globbing patterns (
**) are allowed here too.
Serving your stuff with Jammit
Jammit provides two additional helper methods:
include_stylesheets, that can take one or more targets at once:
By default, Jammit will not package source files into single-file targets in development mode, to ease your debugging. You can change that by setting the
package_assets option to
always. You can still get to non-packaged individual files on specific requests by putting a
debug_assets=true parameter in the request, unless you explicitly disabled that feature in your configuration.
Jammit has a few more options and features. Here’s a full-on configuration example.
Be careful though: while this does considerably reduce your number of network requests, this can quickly push your CSS files above the cachability size limit of mobile devices. So if that’s a concern, tweak your settings until you find the right balance between file count and file size.
There’s just no reason to wade through huge scripts and stylesheets anymore.
About the Author
His current job is CTO of Ciblo, a Rails-centric web agency. He lives in Paris with his wife Élodie.
Find Christophe on:
- Twitter - @porteneuve