No more `grunt watch` – faster builds with the Broccoli asset pipeline
solitr.com
solitr.com
Any transformation step can read in many input files and produce many output files. The built-in dev server tracks the entire asset dependency graph and only rebuilds the assets that are dirtied by a source file changing.
We have a plug-in system, and it's built on top of the same package management system that the SDK uses, so you can get transformer plug-ins as easily as you can get any other dependency.
We still have a lot of work to do to fully flesh things out, but it already does a lot, including supporting complex scenarios like transformers whose own code is the output of a previous transformer.
More here: https://www.dartlang.org/tools/pub/assets-and-transformers.h...
Why does every language need to have its own everything unique to itself?
1. The asset build system is built on top of a bunch of policies and conventions specific to our language's package manager. The build system needs to be able to locate plug-ins somehow, and it's hard to define that "somehow" without some kind of assumptions for how dependencies are located and organized.
2. Like other platforms, we want to minimize external dependencies. If we reuse grunt, then every user has to have node and npm installed. There's nothing wrong with that, of course, and many users do already, but we'd like to avoid forcing that dependency.
3. Since the build system is plug-in based, we need a way for the build tool and plug-ins to communicate with each other. We want that API to be simple so that it's easy to create plug-ins. Since they're sending entire assets through that communication channel, we need it to be efficient too. It's hard to do that in a language-agnostic way.
4. We want to make it as easy as possible for people to contribute to the build system and various build plug-ins. Since we already assume users know Dart, then when those components are also written in Dart, it increases the chances that they are able to pitch in and help.
I don't like walled gardens or feeling like we're reinventing the wheel (though I'll note that before this announcement, I wasn't aware of many other build systems that were as many-to-many based as ours), but there is value in having things be internally consistent.
This first line is a little disingenuous. Technically, it's not backend-agnostic, since it depends on Node being installed on the backend (in the same way that Sprockets [1] depends on Ruby). The Rails asset pipeline is a framework-specific integration of Sprockets. In much the same way, you could more closely integrate Broccoli with Rails if you wanted and call it a new Rails asset pipeline.
The project itself looks great, just the first line was confusing since they started the docs off by comparing apples to oranges.
A better comparison would probably be, "It's comparable to Sprockets (which powers the Rails asset pipeline), but runs on Node instead of Ruby."
The Rails asset pipeline requires a JS runtime, so Broccoli doesn't have any more dependencies than asset pipeline in my mind.
Technically they're right that the Rails asset pipeline does require a JS runtime by default. But it's important to note that the Rails asset pipeline doesn't strictly require the JS runtime; it's only required for the coffee-script gem, which happens to be included by default for new Rails apps. If you're not using coffee-script for any of your assets, then no, it doesn't require a JS runtime.
The main difference between sprockets and grunt / lineman / broccoli, is that sprockets do not watch files. It's a web application that compile your assets on the fly[0][1] when you need them.
So it's lazy, and blocking. Meaning that you do not have to guess if your assets have been recompiled yet. Your browser just ask for them, and get them as soon as they are ready.
[0] In development mode of course [1] In fact it cache the output, but whatever.
Think about it this way. Heroku, for example, can either have Ruby or Node installed on a single server, not both. If you have a Ruby app deployed on Heroku, you could not use Broccoli in production, because it requires Node to be installed on the server (not that you'd want to anyway; it's a precompiler, so you'd run it locally before you deployed to Heroku). The fact that you can have a backend setup on which this could not run means it's not backend agnostic.
Anyway, that's not the disingenuous part. The disingenuous part is implying it's better and more flexible than the Rails asset pipeline, which is an apples-to-oranges comparison. This is comparable to Sprockets. Sprockets does this sort of thing (with a slightly different implementation), but is written in Ruby. Then someone created a sprocket-rails gem, which set up and configured Sprockets to run more tightly-coupled with Rails (making it easier to use within a Rails app). That is what then became the Rails asset pipeline.
You can still use Sprockets with a Rails app, Node app, Python or Django, whatever, as long as you have Ruby installed on the server. Just the same as you can use Broccoli as long as you have Node installed on the server. This tool is comparable to Sprockets, not the Rails asset pipeline.
Now, sprockets, in addition to precompiling, also happens to have a live Rack server that can run in development mode, which you'd have to configure with your app in order to use. In that case, yes, Sprockets has more interdependency with the specific implementation of your web app. But if you just used Sprockets for it's precompiling, and added the Watcher gem, you'd essentially have what I understand Broccoli to be (but I could be wrong as I haven't actually delved much into the Broccoli source code).
I'm not being down on Broccoli. Quite the contrary, I think it looks really cool. I'm just saying that comparing it to the Rails asset pipeline implies that the existing alternatives aren't as flexible or configurable as they really are.
I guess the problem these build tools are facing is the amount that people have invested in Grunt. There are just /so/ many grunt tasks at this point.
Definitely awesome that people are trying to optimize in the devops space but just not sure I could be convinced to switch to something that isn't at least somewhat mature.
With node.js as a host environment, it's easy enough to do pretty much whatever you want as a script/task, and is pretty much cross platform, and works well. Shell scripts come close, but ignore the elephant in the room (Windows).
For the record, I'm biased as all hell, like node.js and JavaScript in general.
I use mingw and make, never had a problem on windows and it's not like mingw is heavy to download. Most people use git and git comes with a bash on windows too, so 0 excuses to learn shell scripts.
Hands full server-side anyway, client-side moves way too quickly to keep pace with the latest and greatest...
Have to wonder why this all requires new build tools to achieve though.
It's obviously technically possible (npm shows grunt-gulp which does exactly what you'd think), so is the grunt architecture so firmly rooted in files that streams could not be leveraged, or is it something that will come in time?
Personally managed to get a nigh on perfect build process using grunt-watch, so no rush to move away.
Nice to see build tools still moving forward though.
You can either do this explicitly (by setting up aliased), or implicitly (by all the temporary file watch jiggery and pokery).
It's unlikely that I will start any new projects with grunt, even though I know it a lot better than gulp right now.
Honestly half the battle is having a great easy to use plugin style that gets lots of people making plugins, and given the incredibly brief mention of plugins which makes them seem quite complex, I'd be quite wary of broccoli at this point.
It manages complexity really well. I have thrown many known failure scenarios at it, and it handled them all without a hitch.
I actually do not believe grunt/gulp vs broccoli makes terribly much sense to compare as broccoli aims to be a accurate/stable/fast build pipeline, it does not aim to replace your task runner. It's primary goal is to be the best possible build pipeline, and should be programmatically accessible to your existing task runner.
Anyways, since I didn't really compare grunt/gulp with broccoli, let me explain what you get:
* primitives make sense, was able to get a fairly non-trivial or ordinary pipeline setup really quickly
* builds so fast, you don't notice them.
* accurately handles changes (deletions and git branch changes tend to often cause similar tools issues)
* doesn't lose changes that occur while building
* true pipeline. eg. describe the transforms, and the system handles the rest. No needing to construct make-shift pipelines yourself
* immediately usable as server (error reporting, locking)
* development builds.
* minified source mapped production builds.
* the Broccolifile API is well suited for constructing custom piplines
And finally:
the above you essentially get for free, if you need customizations the Broccoli file provides are an API well suited for the task.
Now, I wonder if the same approach can be taken in gulp using vinyl-fs? Otherwise, I wonder if it might be worth plugging this tool as a specialized asset pipeline into existing grunt/gulp files.
But.. competition is healthy, I am glad we are getting some great solutions in the space.
> Brunch also tries to do partial rebuilding rather than caching; see section “Caching, Not Partial Rebuilding”
But the end of that section seems to imply that it's still something that needs to be implemented per-plugin:
> Plugins that map files n:1, like Sass, need to be more careful about invalidating their caches, so they need to provide custom caching logic.
I'm excited to see what comes of it, but still prefer the idea of simply running `npm install (sass|jade|less|stylus|coffee-script)-brunch --save`
One thing that has room for improvement though is the syntax, which in my opinion doesn't reveal the intention behind some methods and is a bit too coupled with the implementation. What does `makeTree('lib')` mean? If it's taking a folder and its files then why not rename it to something like `broccoli.requireFolder('lib')`? Also another thing that might improve usability would be chaining compilers instead of calling them directly with the tree as parameter.
These are just minor things anyway, I'm sure the library will improve over time. Congrats joliss, great fan of your work!
Might I suggest lumber.js, considering it's all about trees.
I want to mention mincer[1], which I have used in the past for compiling assets, and it has been an entirely painless process. Definitely take a look at it as an alternative, which has been around longer and has seen assistance from the folks behind sprockets[2] (according to the README) for creating a similar API.
1. https://github.com/nodeca/mincer 2. https://github.com/sstephenson/sprockets
Obligatory XKCD: https://xkcd.com/927/
Prepros is the closest software out there now, but it's not extensible like Grunt.
Engineers will constantly run toward shiny baubles at the expense of everything else.