Sprockets: JavaScript dependency management and concatenation (SvN) | Hacker News Reader

Sprockets: JavaScript dependency management and concatenation (SvN)

37signals.com

36 pointsjakestein21 comments

Sounds like great stuff! I think we shouldn't limit ourselves to source code directory structure so much and this is a step in the right direction.

One problem I foresee with this approach is that it could make debugging harder, if you have to look at the "concatenated" javascript file in production to make sense of what code's included and not, it's harder to know what comes from where than if you just included the original source code files.

One solution would be to have a way to switch between "concatenated" and "normal" modes easily (maybe it's already possible, I don't know), so that normal development would be unimpeded. But you'd have to resort to the big concatenated file for debugging in production.

Another solution would be to somehow preserve the information about what comes from where, so that via a browser plugin you could browse the concatenated file as if it were the original modular files.

Lately I'm getting really interested in ways to preserve information about "what comes from where" so that you could point to something in a generated output and ask the system to explain the sum of all inputs and considerations (decisions, transformations) that resulted in that "atom" being generated in the output through the various "semantic levels". You usually generate a low-level representation from a higher-level representation and it can be useful to proceed carefully from one transformation to the next so that you preserve meaningful semantic information about intermediate stages.

Unless I missed something, this is exactly what django-compress already does for the Django framework:

http://code.google.com/p/django-compress/

We just started using it in a project I'm working on. Most excellent.

Thanks! I've been meaning to write this exact thing for a while but never got round to it, now I don't have to :)

Cool, django-compress looks similar to the sprockets-rails plugin. A few differences:

django-compress makes you define your source files and concatenation order in an array in a configuration file. Sprockets lets you define your dependencies with annotations in the source files themselves, meaning plugins can have their own dependencies and integrate themselves with your applicaton's build system.

Sprockets doesn't do minification yet.

Sprockets doesn't do CSS concatenation, either. But it does have an asset provisioning feature, where plugins can specify a collection of files (e.g. css, images) that should be copied into the document root. (Rails itself provides basic CSS and JS concatenation.)

It's nice to see other similar but independent implementations of the same idea.

I dislike the comment "require" directives way of specifying dependencies. They should be real JavaScript function calls, which could either be analyzed ahead of time and stripped, like Sprockets, or used with a JavaScript loader system at runtime. This would remove the CGI/Ruby dependency and make the whole system more reusable (for example, server side, where concatenation isn't necessary/desired).

Respectfully: no, it shouldn't. :) There's plenty of libraries that do this already. Sprockets solves a different problem.

Can you elaborate? I assumed the problem was "JavaScript dependency management and concatenation".

What I described is essentially what we do in Objective-J. During development it just resolves the dependencies at runtime, but when you deploy you can flatten them into one big file.

I love how the options are Rails Plugin or CGI. If they write their marketing sites in PHP, why didn't they go with PHP instead of the CGI route? Or, is it the case that the CGI was just a "hey, you don't need to use Rails to take advantage of it" ploy?

EDIT: after reading the CGI source, they use some mod_rewrite trickery to avoid running the CGI if theres a cached file.

I originally wrote Sprockets 1) as a replacement for Prototype's build system and 2) as a library that could be used within Rails. Just before releasing 1.0, I added a simple CGI wrapper around the library so we could use it in our marketing sites and so people could use it relatively easily without Rails.

I like the idea of distributing a PHP wrapper. I'll try to add one for the next release.

And there's no "ploy." Interesting theory, though.

After finding that the CGI was smart, I can see the value in it. The "ploy" comment was written "pre"-enlightenment.

Wonder how it interacts with browser cacheing and parallel JS resource downloads?

JavaScript resources are never downloaded in parallel. They are downloaded sequentially because they are interpreted while they download, and (existing) browsers always run JavaScript in a single thread.

This will have very little effect on caching. You will still need to configure your web server for optimal caching, by setting the Cache-control and Expires headers, for example.

Even so, this still has two main benefits: The JavaScript is minified, which will reduce the amount of raw data sent to remote users. Plus, by concatenating all the files together, you avoid the need for multiple separate HTTP requests.

Incorrect. In the latest browsers, js can be downloaded in parallel, but executed in the correct order - sequentially.

AFAIK, that includes the latest safari, firefox, and IE8.

However, it's still obviously a good idea to concat and minify all your scripts.

Cool, thanks for the tidbit.

I had previously been told that a JavaScript download basically halts the entire processing of the remainder of the page -- including any downloads that might occur there (CSS, images, and more JS) -- because the browser can't start parsing the rest of the page until it has interpreted the JavaScript. That's why it's best to move your JavaScript down to the bottom of the page.

Is this just the way older browsers used to behave? Are there still any restrictions like this that I should be aware of?

My rule of thumb with javascript is to put everything that is critical to page functionality right at the top and everything else as far to the bottom as I can (just before </body>), that way all your buttons/sliders and other active components will work as soon as the page is rendered and any other javascript will never hurt performance.

There is a page describing which browesrs support which parallel download types at http://stevesouders.com/ua/report.php

It seems that concatenating scripts could hurt performance in browsers that can do parallel script downloading, since it can no longer take advantage of the parallelism.

It's be an interesting one to test, although I'd expect the gain in compression is worth it.

eg gzip(concat(A, B)) compresses to a smaller file than gzip(A) + gzip(B)

For example, if I take 20 js files from Mibbit and gzip them individually, total size is 47,890 If I concat then gzip, it's 42,681

We use ant to accomplish the same thing.

Yeah, I was gonna say "make" combined with some new tool to read the metadata from the js files would probably be doable.

We are using make for exactly that purpose, works like a charme.

Reply on news.ycombinator.com