Rolldown: Rollup compatible bundler written in Rust
rolldown.rs
rolldown.rs
Working in such an environment is a dream. Need to add a feature? Chances are that someone out there has already built every piece of it and all you need to do is track them down and stitch them together.
1) esbuild isn’t good at splitting 2) rollup is too slow
Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance? Why is it the best option to introduce yet another tool?
Rollup is written in javascript; not only is the runtime slow, the multicore story is terrible.
Improving esbuilds splitting functionality is hard because splitting is full of hard tradeoffs; the spec allows module imports to have side-effects and requires that they be evaluated in order. It's therefore impossible to do optimal code-splitting without violating the esmodule spec (potentially introducing subtle errors), and the `esbuild` project values correctness even more highly than performance.
Personally, I'd be very happy with "If you enable the 'make it fast' option and have modules with top level side-effects, you will get weird bugs", but I respect where they're coming from in not wanting to do that.
Is it fair to say that it was a bit of a sales pitch?
When key components of the ecosystem that were written in the language of the ecosystem itself are being rewritten in other languages - I think it’s hard to find a more compelling evidence that all is not as great as it was advertised.
I am sure no-one ever claimed that things (properly) written in JS are as fast as things (properly) written in C, Go, or Rust, and running in a native environment.
JS on the server may have been faster than php or python, due to the non-blocking nature of the event loop; but even that may not be the case anymore.
Generally, the pitch has been that js is "fast enough" for most tasks.
a) It's actually calling some native code internally (i.e. rust, c/c++ or V8 internals)
b) JIT'ed
Path "b" is very hard to achieve in CLI tool
Why say that?
Yes, perhaps if the CLI tool is being invoked many many times in succession.
But otherwise JITing isn't an issue: either (a) your program runs quickly in which case the Ignition interpreter is used and the JIT is not used, or (b) your program runs slowly in which case JIT (Sparkplug/Maglev/Turbofan) has time to warm up.
Each execution will be a fresh VM though. A CLI tool isn't a long-running daemon. There is also the whole lack of parallelism without forking in this ecosystem.
Yeah. It’s in the order of 4x slower than Java for most things, but it’s also like 100x faster than Python.
It’s all a matter of perspective, but in general, JS is probably about the slowest language I’d be inclined to reach for for long term solutions.
I get that VSCode might not be performant for others. I'm on a M1 Mac with 32gig of ram. I never notice any slowdown compared to any other editor I've used. I'm not saying other editors aren't measurably faster. I'm saying I don't notice any issues personally.
Maybe there are some considerations I don't know about but it does seem like hard forking esbuild would be much faster way to get there (hard fork so that they are not blocked or slowed down by whatever esbuild owner wants).
I would almost surely give up before understanding what esbuild does and how, and what needs to be modified. (Because Go basically gives me the creeps.) So someone with a lot of Rust experience probably simply opts for the rewrite.
That said, it seems folks on the Vite team decided to give this a try because of oxc_parser (a JS parser written in Rust), so probably it's not because of Rust-Go sentimentalism.
// Though ... there's also otto[1] (a JS parser in Go) ... so who knows! :))
// And of course even more context here[2] and here[3] (this one is from Evan You)
//
[1] https://github.com/robertkrimen/otto
I tend to think Rust isn't suitable to rewrite JavaScript tooling, any compiled language would do, or even native addons, but apparently dealing with borrow checker for userspace software that doesn't require 1us performance within 512 KB is appealing.
Currently watching https://www.infoq.com/presentations/rust-javascript/
> any compiled language would do
So why not Rust?
Didn't realize it was a self bootstrapping language now. Thats awesome
Similar performance levels can be attained with any AOT compiled managed language, with much better developer productivity.
Either one needs to deal with the borrow checker for performance that matters, or if the performance doesn't matter to put .clone() all over the place, then what is the point beyond CV building?!?
This leaves you with a fast language, with robust and relatively easy multithreading, and a rich ecosystem of high quality libraries. I don’t know any other AOT language that can come close to such productivity. The closest may be golang, but it’s not ideal for parsing.
Multithreading is just as "relatively" easy in other compiled languages.
Golang for example does not enforce use of synchronization when mutating shared objects from multiple threads, even though it can lead to crashes and vulnerabilities, just like in C and C++.
It may be hard to appreciate Rust's fearless concurrency until you see for yourself how it can save you from heisenbugs. You can invoke any function, which could be even a 3d party dependency using more dependencies and touching a million lines of code, and Rust will tell you right away if any of that code has some thread-unsafe behavior (e.g. some method may have internal cache without proper locking). With the safety net of the borrow checker you can use constructs that would have been considered irresponsibly dangerous in other languages, like spawning threads referencing on-stack values of other threads.
Writing multithreaded native applications is my bread and butter. Including enterprise grade application servers with multiple clients and various IT systems accessing it. I can't recall when was the last time I had any problems caused by improperly mutating data.
The quality of the tools one uses matters a lot.
https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-I...
The fact that Mozilla can not do a particular thing in particular way has zero value for others not in the same boat. I am not Mozilla. Do not know their problems and do not care. Maybe parallel CSS is too complex by nature. I can only relate to my own situation. My commercially deployed systems work and serve bazillion of clients with no problems.
I've also written systems in Rust, some embarrassingly parallel and some that are more complex than that. The embarrassingly parallel ones never took more than 10 minutes of effort to do. The more complex ones are harder but still possible.
I agree that your data point should also be assigned non-zero weight.
It is well above zero. But this is for very specific situation - parallel CSS.
I on the other hand had never had problem managing / mutating state in my multithreaded programs. I did have one situation with deadlock when interacting with Directshow signal processing. That was around 2002 I think. Took me one day to analyze and fix the code after the bug was reported. But Directshow is an incredibly complex piece with some weird behaviors. Other then that I do not recall any real problems. Btw that particular software was desktop application and written in Delphi, not even C++.
So yes to me the opinion of Mozilla's team is totally irrelevant. It is of course totally mutual but I suspect that my situation - C++ application servers is way more common than parallel CSS.
> Multithreading is just as "relatively" easy in other compiled languages.
Maybe that is true for your case, but broadly speaking it isn't true.
Having worked on large web-apps, I'll take all the speedups I can get, thanks.
You specifically seemed to be saying "Why use rust when you don't want top tier performance", to which my response was "I do want top tier performance."
That aside, with regards other languages I'd consider candidates for high performance tooling, I'm either not proficient with them, or I don't enjoy working with them.
Undoubtedly, language tooling being fast improves your ability to iterate. 1 second vs 60 seconds doesn’t “seem” like much, but when you have that speed, it becomes very difficult to give it up, and when you know it can be faster, it is incredibly frustrating when it’s slow.
Although I suspect that as your domain knowledge of the software improves, the effect flattens out.
- you are a c/c++/low level guy who knows how allocation works very well and figure it out
- you are a java/c#/python guy who wants to get going and use clone() a lot
Either way, the performance that you get is worth the troubles.
You can get similar performance with any AOT compiled language, even managed ones.
If only you could have warned all the people that the work they did was untenable before they went and did it.
What specific languages do you have in mind?
*and C and C++ but writing tools in them is like self-lobotomy.
EDIT: And Dart (which is not that mainstream).
C is a lower level language than Rust so sure it is harder to write complex tools. Modern C++ - writing tools is at worst as easy than Rust and way better in practice because of a huge amount of libraries to suit every taste and C++ being more expressive / supporting more programming paradigms.
Statement like yours are not doing any service to Rust. Rust advocacy can do very much without such "help". Hopefully it does not represent culture of Rust community in general.
When I wrote that statement, I didn't really mean C++ the language, but C++ the developer experience. I have tried to write a few command-line utilities in C++, and while the language proper is quite capable, dealing with CMake, package management, header files, and testing is just more headache-inducing than in Rust for me. Most of it probably stems from my lack of experience with C++, but it doesn't change the fact that despite C++ being 20 years older than Rust and having an order of magnitude more users, most of new JS toolings are not written in it.
Feel free to prove me wrong by pointing me to good tooling projects in C++. I actually want to learn more C++ best practices for working with another language.
I can't point you for particular tooling C++ project. The only 3rd party source code I use are some C++ libraries. All it takes in my particular case is adding couple of lines in CMakeLists.txt file and I am very far from being CMake expert.
The documentation is massively verbose but doesn’t really say anything helpful. There’s basically no “Modern CMake” examples that anyone can agree upon. (I do see that there’s some GitHub repos these days that maybe do show some of it). There’s very little information on “The CMake way of doing things”. You really need to learn by examination, and your assumptions in the end will very likely be wrong.
Even just adding a package fetch in CMake is infuriating. What is the currently accepted way to do this? (I admit that I haven’t used CMake in a while, but as of a year ago, the internet and CMake docs did not agree on the correct way, even internally)
>"Even just adding a package fetch in CMake is infuriating."
I am not sure how to do it because I do not let build process / environment fetch things from the Internet. I always download / deploy / update manually upon real need. I would agree that having IDE do it for you is convenient but since I only use very few libs there is no need for me to do it at all.
If I need to create new project I just copy a template, fire up CLion IDE and it all takes few seconds from the - I am going to do it, to start actual coding.
Cargo/Rust and Go show how compiled languages can have drastically superior development experiences. CMake is primitive in comparison. Its only upside is its flexibility, which I reckon isn't worth it for most cases.
It took me about the same amount of time to write few dozen lines of CMake (to get build working with all the external libraries), as it took me to write few thousands lines of C++.
I wasted way too much time dealing with CMake.
I claim BS unless those "thousands lines" were cut'n paste.
Well thank you. I just trying to guess where did you get the idea that I am not using 3rd party libs.
>"C++'s lack of a proper package manager means we're all reinventing the wheel because adding dependencies is non-trivial."
I think it is trivial. Here is particular example for cpp-http library that I use in some of my projects:
a) download from github
b) add add_subdirectory("./3rd-party/cpp-httplib") into CMakeLists.txt file
Additionally, you get a nice list in a simple spot of exactly what you need that isn’t just documentation. Anyone can easily pull your project and contribute.
Being able to easily manage deps has issues (such as a tendency to trend toward left-pad), but there’s also significant benefits as well that become readily apparent when you compare cargo with CMake (I understand they’re not completely comparable)
That being said, Cmake is a far cry from the ease of use of cargo (albeit with potentially much more flexibility).
Ugh. You’re disagreeing about which languages are best for some task. Are either one of you being-an-advocate? That’s up to interpretation.
It’s not fair to say that someone who vouches for Rust is being-an-advocate while someone who vouches for C++ is just arguing normally. What basis is there for that?
I could say that you are not doing any service to Rust-naysayers. That I hope that the culture of Rust-naysayers are not represented by people like you. Would that be fair?
As the main author of cargo-nextest, I'd challenge you to write a tool with a similar complexity and quality level in C++. I got so much leverage from being able to use the Rust ecosystem, including being able to port to new platforms with literally zero code changes. (Be sure to get the signal handling exactly right.)
Yes it is. I love the borrow checker.
But also yeah, Rust is just a really nice programming language, I prefer to write Rust so obviously I'm going to write software in Rust and that's likely true for others too.
Plus, on a personal level, using a language with pattern matching and algebraic data types makes writing tooling for parsers and such much more ergonomic than in languages without.
However, doesn't the explanation given for why the Go version was 10% faster mean that esbuild was built to take advantage of a fixed number of cores rather than all of them? Sorry if this is a dumb question, I'm not really that experienced with parallel computation
Thinking about the problem, I think that at the very least parsing could be parallelized. Assembling everything into one output might not be parallelizable. But I haven't looked at what happens in esbuild.
I do know that it's fast enough that when I switched from webpack I had to check that it actually did something, because it returned immediately.
Not being glib: you didn’t mention why you brought that up.
I have no authority and will not stand behind these claims if challenged.
I want this on a shirt
1) I know Rust, not Go (esbuild is written in go-lang)
2) I don't have to worry as much about breaking Rollup, and I'm not nearly as confident in my ability to write performant JS as I am in writing performant Rust.
If you can reduce the problem to a series of operations on pre-allocated typed arrays, JS is fast enough.
Unfortunately in the case of Rollup that would be hard to do.
In some respects, rolldown is also the next generation of Rollup[0] and is a direct answer to rspack[1], which is a webpack compatible rust based toolchain from ByteDance
[0]: Which switched to SWC as its underlying parser from acorn, which makes me wonder if that influenced the decision to use OXC, perhaps they found SWC to be troublesome. I know the Rollup and Vite teams are extremely close, and wouldn't be shocked if any troubles they had with SWC weren't shared with the Vite team when considering underlying technologies
From the start you want to use x.y.z tech then the justification is meaningless.
I've found esbuild is pretty good at splitting when source formats and outputs are ESM. (Given the name esbuild a focus on/priority for ESM input/output makes sense to me.)
Is this maybe Vite team saying that they still have too many brownfield CJS dependencies and still don't believe in ESM output for production bundles in 2024?
I think the bigger problem, from my experience here, is still the question of not moving to ESM as the default production output. That seems out of date.
Personally I feel like moving a JS tool to rust means 99% of JS/TS programmers are unlikely to be able to supply a patch to fix a bug or participate in most ways. I suspect some people will consider that a good thing though.
For Rollup: the Rollup team itself has been trying to incrementally improve Rollup's performance, e.g. by swapping acorn with a Rust-based parser. But there's only so much you can gain starting from a pure JavaScript base, especially considering multicore utilization. Another aspect of the performance is in the back-and-forth between Rollup (on the JS side) and native transforms (swc, esbuild) - there is a lot of overhead repeatedly parsing / serializing ASTs and then passing strings across JS/native. By building on top of Oxc (which will ship transforms in the future) we hope to be able to stay on the native-side as much as possible to avoid such overhead.
I only tried rollup from the beginning because the ES6 project I was trying to bundle suggested it.
With rollup, I can just ship it as a web extension and it’s still 100% readable.
In web extensions you can’t quite use sourcemaps unless you include the sourcemap in the .js file or upload it to your own server
JS tooling is mostly single threaded. It is expensive to stringify a large AST and then parse it again over IPC vs just transferring an object (which is not allowed by runtime). This cost incentives more things happening in the same process, so there’s a limit to how fast you can be with JS
It went the way that many one-man projects with no community backing so, which made me sad. I moved to Rollup, and then recently moved to Vite to get back on the instant HMR train.
Thanks for your work in this space, steelbrain! pundle was awesome!
Is there a way to enable this in Vite the near term to test?
If you're doing all that, you could probably support rollup plugins as well, which it sounds like they want to do.
In some respects, rolldown is also the next generation of Rollup (potentially) and is a direct answer to rspack[0], which is a webpack compatible rust based toolchain from ByteDance
I have used SWC internals recently for a toy project of mine and was generally let down a bit by some of the rather unwieldy internals, insufficient documentation and pretty hefty footprint.
I also blame Vercel, they have put zero investment in documenting SWC. The person behind the project used to be more prolific in documenting things, and that all stopped once Vercel scooped them and the project up for use in turbopack (which has not materialized as a standalone solution like they promised, funnily enough)
Vue projects also use vitepress
Astro, Jekyll, Hugo and others always were just a bit too cumbersome for my taste.
For my own (edit: as in personal) projects I tend to just “dev in build mode” with some subset of that bolted on, because I’ve been burned so many times by “works in dev” issues.
Why do I even need to bundle a nodejs api? My current project's backend is completely in Rust because I actually found that easier to deal with. Kubernetes was easier to figure out for me.
You don't need to bundle server-side code unless you're using a cloud provider that limits how much source code you can upload or some other limitation like that.
More importantly, how is this being funded? I find it hard to believe that core team is running off sponsor dollars. Is this project going to be around next year? Maybe it's coming from Vite sponsorship backers.