Hacker News Reader

Comment by dwohnitmok | Hacker News Reader

Looking at the TiKV compile time graph, if that's for 2 million lines of real-world (not synthetically generated) code, that actually seems kind of reasonable.

~12 min for a full release build? ~5 min for a full dev build? What looks like several seconds for an incremental build?

I suppose it's good that a (former?) core contributor and large user of Rust holds the language to such high standards, but this doesn't seem especially bad.

I'm kind of curious how much faster compiling a large Go codebase is. How fast, e.g. does the entirety of Kubernetes compile? I'd imagine it's probably under a minute, but is it several seconds?

View on HN

> I'm kind of curious how much faster compiling a large Go codebase is. How fast, e.g. does the entirety of Kubernetes compile?

I was curious too, so I just tried it. On my 6-core previous-gen MacBook pro, I get 2m18s. That's for SLoc = 3296486 (assuming all deps are vendored and non-deps are not-vendored, which I think is true), so about 16K SLoc/s.

Or to put it another way, if Rust compiled as quickly as Go, we'd expect to compile a release build of TiKV in 2m.

Can you see how long incremental builds take (under a minor change)?

it was very slow, almost a minute! but it has a custom build script, that looks like it’s not really made for incremental builds? it looks like there’s also bazel support, which i would guess is much faster, but i couldn’t figure out how to get it to work...

It would be interesting to have a `reverse` blog post "why Go compilation time is fast" e.g. which optimizations are overlooked/done quickly by the go compiler ? Could the binary quality be improved by increasing the compilation time ? AFAIK you don't have optimization flags in go (or the default one are optimum).

And to be honnest, I have no idea what the compiled Go code looks like. As much as I have no idea what actual instructions are executed by the python interpreter.

Well one big reason is the biggest feature people are complaining about: A lack of generics. According to [1], there's a fundamental trade-off with respect to generics:

"The generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?"

At the moment, Go has no generics, which translates to "slow programmers", but not "slow compilers". It's likely that the simplicity of the Go language as a whole -- which leads to many complaints -- is a major factor in the fast compile times.

EDIT: Lest it seem like I'm bashing Go here, the quote above is from one of the core Golang developers (in 2009 no less), and is mirrored in this from the proposal [2] from another core Golang developer. It's meant to be a shorthand for, "Generics do actually save programmer time and effort, and Go programmers are working harder than necessary because the language doesn't have generics yet." The point of this post is to point out that generics have a cost (slow compilation time) and that fast compilation time has a cost (no generics -- at least, not without inventing a new way of doing generics).

It's possible to love something while still seeing its deficiencies and wishing its improvement. In fact, I'd argue that's the only way to truly love anyone or anything.

[1] https://research.swtch.com/generic

[2] https://go.googlesource.com/proposal/+/master/design/15292-g...

> At the moment, Go has no generics, which translates to "slow programmers", but not "slow compilers".

Also note that one of these assertions is measurable, and the other an unsubstantiated opinion.

What I don't like about the "generics dilemma" framing is that the problem exists regardless of whether your language has generics or not.

Here's what I mean. Say you have a Vector class that can operate on ints or floats. You could make that a generic, in which case the compiler can either (a) duplicate the code for each type (monomorphize) or (b) do dictionary passing and get slower runtime. But if your language doesn't have generics, you have exactly the same problem: you as the programmer must (a) duplicate the code for ints and floats or (b) use an interface and get slower runtime. Not having generics doesn't solve anything. It just means that you, the programmer, have to do things that the compiler would otherwise do for you.

Although, one interesting aspect of the trade-off here is that a programmer who manually monomorphizes only has to do so once(^) -- their effort is reused across multiple compilation runs -- whereas the compiler normally has to monomorphize on each compilation run.

(^) Of course, you then have the burden of keeping multiple monomorphized implementations consistent when you make a change that needs to apply to all of them.

Monomorphization tends to happen "on the fly", and even if it wasn't, it would still be cheaper than duplicating all of the work of parsing and type-checking (which is needed if the user manually monomorphized).

That can't be right though. Java and Kotlin have generics, fast programmers, fast compile times and as generics are erased there's no more binary bloat than without them.

I think the assumption implicit in that statement is "generics over value types compiled AOT". But that's not the only way to do it.

But slower than optimal execution times! Since everything is boxed.

For generics over primitives, yes, but the JVM only has a small number of those so they can be easily hand specialised:

http://trove4j.sourceforge.net/html/overview.html

The rest of the time you're dealing with references anyway, so there's no boxing.

Well, sure, in the context of a language that doesn't really have value types and is already doing everything w/ dynamic dispatch, you can have "zero cost" generics, in the sense that there's no code bloat and no perf penalty, vs what you already had.

But that's because your language is already leaving some performance on the table!

If you want a language that's as fast as possible, you want something like

    GenericContainer<Foo> foos = ...;
    for(var f in foos) f.DoSomethingFooish()

to be able to transform into a contiguous chunk of "Foo"s in memory, that the loop is traversing, and doing no dynamic dispatch (and potentially inlining!) in each of the "DoSomethingFooish" calls.

I don't think you can have a generics system capable of achieving that level of performance w/o also brining w/ it the downside of more code generation & extended comp time.

(P.S. Also, Java is getting support for user-defined value types, right? How will those interact w/ the generics system?)

Yes, that's a good point. I'd counter though that in C++ and similar languages value types and memory management get conflated in ways that hurt performance. Java has a really, really fast heap and allocations get laid out contiguously by the GC in ways that have a measurable + significant impact on cache hits and performance.

In C++ you see std::vector with large-ish values all the time, even when it doesn't really have any memory layout justification because that way you get semi-automatic memory management and with pointers you don't. This can easily lead to large amounts of pointless code bloat, hurting icache hit rates, compile times, binary sizes and more, even in cold paths where memory layout is the least of your concerns.

Not sure yet how generics in Java and value types will interact. There have been some prototypes of generics specialisation so it'll probably end up like C++ but, hopefully, with way less use of value types - restricted only to places where they make a real improvement. That'll be a lot easier to measure in Java because as long as you stay within the value-allowed subset of features you will be able to convert between value and non-value types transparently without rewriting use sites. So you can just toggle it on and off to explore the tradeoffs between code generation and pointer indirection.

Ada, D, Delphi, Eiffel, .NET Native, Ada all compile quite fast and have generics support.

Is this with gccgo or gc? Compiled code quality is worse for gc, though that is mostly apparent on CPU bound code that most likely won't be written in Go (but might be written in Rust).

Yeah those build times are insanely good for a 2 million line project. I suspect, however, that the codebase makes judicious use of generics.

A clean dev build for one of my more experimental projects that uses generic heterogenous lists takes five minutes to compile for fewer than two thousand lines of code and a single test case. In that project, however, I'm pushing Rust's type resolution algorithm to its limits (basically as a giant constraint solver looking for a unique solution).

Our ~1M line of C++ codebase take around ~15 mins on a "full" [1] rebuild, and that's with the support of a custom distributed build system. That's bearable considering that running the unit tests takes again as much.

12 mins for a full build of 2M lines of code on a single box is quite reasonable, and it is great if they are pushing for better performance.

[1] because of aggressive caching if is hard to really do a clean build, the closest you can get is touching one of the 'god' headers included everywhere.

Yeah, I'm thinking back to an unnamed game engine that touching any of the core headers incurred a ~1 hour rebuild, ~30 minutes with incredibuild.

Heck, if you've ever had a complex codebase with LTCG, I've seen cases where linking alone took ~25 minutes.

Does it start with Un and end with real Engine?

There will always be applications for doing a fresh complete build. Like, searching for a bug in different code revisions. You're just not doing that if each compile takes 5 minutes.

As a baseline, a non optimizing compiler for a simple language should be able to do 1 million lines of code per second. Of course, most languages are not simple.

Incremental compilation is an essential operation. I probably do it more than 100 times a day. Just like editor responsiveness, it can almost not be fast enough, and if it takes too long it can bring me out of the flow. I would say that over 0.1 seconds any speed improvement is welcome. More than 3 seconds is definitely a nuisance. More than 15 seconds is extremely frustrating when dealing with certain kinds of code.

I'm curious, are there any 1 million LoC projects out there that compile in under a second?

Sure. TCC (the tiny C compiler) runs at over 3 million LoC/s on my machine (on a single core!) and GCC debug builds aren't that far behind, so for C (or very orthodox C++) projects it's doable. I keep my own C++ codebases under an arbitrary target of 10s for fully optimized release builds and over half of that is wasted by the NVCC.

Are there any 1 million LoC codebases compatible with TCC that can be run in under a second with it?

I ask because there's various unexpected things that can make large codebases compile more slowly out in the wild that don't show up in smaller codebases (as well as non-linear scaling of certain components), often making simple extrapolation of how quickly a smaller codebase compiles to a larger one inaccurate.

One million lines is a good if arbitrary point where a lot of those can be sussed out.

I don't know the answer to your question, but with regards to the million LoC/s target it's worth noting that an AMD 3970X can compile the Linux kernel in under 24 seconds [1] (that's an optimized GCC release build), which is around 28 million LoC. Even though I'm sure conditional compilation will cut some of it out, it's close enough to be in the ballpark.

[1] https://www.phoronix.com/scan.php?page=article&item=amd-linu...

I don't have concrete numbers but I keep hearing OCaml's compiler is extremely fast.

> You're just not doing that if each compile takes 5 minutes.

I've bisected large codebases for bugs several times where compiles take hours.

Yeah but I bet you wished it took less time.

Even when bisecting you can still reuse the built dependencies, at least if you're using sccache.

The only relevant info I could dig up was this issue from Kubernetes in 2016: https://github.com/kubernetes/kubernetes/issues/27444

It seems to be discussing the speed of building Kubernetes on their build-servers, targeting multiple architectures at once. Unfortunately all the build servers are internal to Google so I don't know exactly what's being included (or what's being run), but this comment (https://github.com/kubernetes/kubernetes/issues/27444#issuec...) details the breakdown of time spent:

    30m of build
    7m on cluster startup
    12m on tests
    7m on cluster teardown

So, taken with an enormous hunk of rock salt, it seems like Kubernetes might take around 30m to build on a cluster, to build (possibly multiple) release artifacts. Seems in line with Rust.

What’s the state of parallel builds? We’re getting 16/32 core cpus. In a few years, we’re all going have Threadrippers.

Of course everything can’t be done in parallel.

Almost all parts of the compiler (at least for LLVM) are still strictly single-threaded. You can't even use threads yourself if you write your own LLVM module pass.

Some of the parts of a compiler are inherently serial. e.g. parsing a file. But if you have to parse multiple files, then hey presto, parallelism. The linker has to work as a spooler for writing the resulting executable or shared objects, but afaik, there's no reason for symbol resolution to not be done in parallel. And if you want to optimize your GOT then that could also be sorted by a page rank style optimization which is the algorithm which was the foundation of the phrase 'embarrassingly parallel'.

Given that most sweet shop ITs provide i7 and i5 with 8GB and HDD, good luck getting that hardware adopted there.

That's compiling for 10 os/arch combos, so ~3m per.

That makes a lot more sense. Go was specifically designed to compile fast.

On the other hand, these are running on some big cluster with a whole bunch of CPUs, so I wonder if you’d still get 3m per build on something like a laptop?

Any individual build they're referencing there runs on a single vm (the cluster referenced there is the k8s cluster being built up and torn down on that vm to run e2e tests on). It looks like at the time they were using GCP standard-8 instances, which are 4 core/8 thread w/30GB of ram. It's hard to say exactly what the current build time is, as it looks like they do some build caching now; on my desktop a fully clean build of current kubernetes master (3.2 million loc) for linux/amd64 takes just under 2 minutes.

Shhhhhh!

If we keep complaining about Rust build times being "slow" they'll keep making it better!

Yeah, I have a C++ project a fraction of that size that takes longer to compile. I'm quite happy with Rust :)

Glad a core contributor has such high standards though.

Is it template heavy? One thing I've noticed code switching to Rust from C++ is that I can write generic heavy code without seeing massive delays in compile time. Conversely in C++ I'd have to create source files with explicit instantiation to get incremental builds of my template usage which is a pain.

i love scala. it would not only be longer but you’d have to compile it 3 times for each major version!

So you're not cross compiling your code for Scala JS (0.6/1) and Scala Native as well meaning 6 full compile cycles ?

Shame on you.

Linux kernel takes ~60 seconds on TR hardware according to phoronix, SLoC is what ~15-18 million

Most of those speed compile benchmarks are done with "make tinyconfig" -- where a small fraction of the kernel is being built (a few million SLoC at most). A kernel build using "make allyesconfig" takes significantly longer than a minute.

A clean build of the Linux kernel can be very fast, but the config is very relevant. A custom build specific to your hardware will touch only a small fraction of the overall kernel tree, whereas a generic build including a large swath of hardware will take much longer.

(AFAICT there's no easy way to measure the LoC actually compiled - but one rough way to estimate it would be to take all the .o files listed in the build process and count the LoC in the corresponding .c files and the .h files they depend on).

> AFAICT there's no easy way to measure the LoC actually compiled

I take it you can't just run it through the preprocessor then count the lines of code that are output from that?

The kernel is composed of a bunch of independent source files which share headers; thus, running them through a preprocessor would massively overcount the LoC. (In theory, this would get you the “actual compiled LoC” but that’s a useless metric because most header files are just declarations that are easy to parse and “compile”).

Easy way is to grab the debuginfo files, run it through addr2line and count.

Eh, I was wrong on Epyc a make defconfig && make takes _16_ seconds but I don't really follow what is actually in defconfig so it might be bare bones.

https://www.phoronix.com/scan.php?page=news_item&px=Linux-16...

Compare to swift it does look pretty reasonable.

It's interesting seeing these kinds of compilation times being "long" from a machine learning perspective.

The equivalent to compiling code in machine learning is training a model. Even on good hardware you can spend hours training a single model. Some of the really big pre-trained models like BERT can take days being trained on a farm of top-of-the-line purpose-built GPUs, which is why people almost never re-train them from scratch without similarly huge amounts computing power and very specific needs.

The equivalent to compiling in machine learning is compiling :). You just ALSO have to train your model.

As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know. Kinda like the pervasive "but isn't Java really slow?" thing that still doesn't seem to have died either.

Edit: yes I know this article was written by someone who knows what they're talking about (and as they're a steward of the project I understand why they're calling this out for improvement, and somewhat exaggerating how bad it is). I was talking about the occasional comments I see on hacker news about how people aren't even bothering to try rust because compile times are "unusably" bad. 5 mins for a dev build of 2 million lines of code is not anywhere near reason to not even bother trying the language. No personal project or learning scratchpad is ever going to come close to 2 million lines. And it's not worse than many other languages that are used for large projects.

I don't necessarily disagree, but this was written by one of the original authors of the Rust compiler, so I think he knows what he's talking about.

I have a few beefs with this article, but to suggest Brian doesn’t know much about Rust is just plain incorrect. He is literally the #2 all time (human) contributor to the project: https://thanks.rust-lang.org/rust/all-time/

Well if you read the article you would have known the author is one of the co-founders of Rust.

I think that it depends on where you are coming from. For Go, C#, Java devs and for those coming from dynamically typed languages, the compilation times will be unbearably slow. C++, scala, swift developers are probably used to slow compilation times.

C++ can be relatively fast when making use of binary dependencies and having a modular build.

This is the one place where Rust differs from many other languages: Dependencies and libraries are typically brought in a source-dependencies and built with your own code.

This typically makes the first build of any project considerably slower than for other languages, since you have to also build all your dependencies, but I think it allows for a really simple and predictable way to work, not to mention support any kind of architecture, across all packages.

It's obviously a trade-off, but I think it's a trade-off which is clearly worth it.

A trade off that requires buying a compiler rig, it is almost unusable to compile Rust stuff from scratch on my travel netbook.

If I plan to do some Rust coding on the go (no pun intended), better do a full build at home before packing.

A 5m C++ build turns into 30m Rust one, for the same project, ported across languages.

Not to mention that it means on a large team project everyone one is compiling the same stuff over and over again, given that cargo does not yet support code cache servers.

Yes there are some workarounds like sccache, but they are extra stuff one needs to install.

> Not to mention that it means on a large team project everyone one is compiling the same stuff over and over again,

That's a bit dramatical. You typically only compile your dependencies once during the project lifetime.

If you on your netbook have done a initial full build, you'll only get incremental builds from there on, just like with C++.

But unlike C++, dependency management isn't hell, and you are almost always guaranteed to have your dependency supported on the target you are building, OOB, without any fiddling or setup.

It's a trade-off, but I definitely think the Rust-team made the right decision here.

Dependency management isn't an issue when using OS packages, and Conan also does binary dependencies.

Compiling from source also does not solve the problem that a crate author hasn't taken a specific OS into account.

Yes I am being a bit dramatic, but this kind of issues do impact adoption, and there are many industries where shipping source is just not an option.

> Dependency management isn't an issue when using OS packages

Come on. What do you find most developer friendly?

1. "cargo build && done"

or 2. Try to uncover what dependencies this project really has, and then proceed to map out what the packages (and dev-packages) for those dependencies are called on the linux distro you are using (debian, ubuntu, fedora, arch, etc), not to mention what they are called on your specific version of that distro, root up, install all that stuff.... and then try ./configure yet again?

> but this kind of issues do impact adoption

Indeed. In 2020 I wouldn't bother adopting any kind of language which prefers the latter flow to the former one.

"cargo build && done" only works if every crate author has taken my OS into consideration and not used OS specific API or file locations.

And it is more like "cargo build && off to lunch".

I am usually on Windows, and most commercial vendors nicely sell us their already compiled binaries. No need to hunt for anything.

Regarding Linux distros, if it isn't on the official repositories, Conan provides exactly the same experience as cargo, only faster because it supports binary libraries.

And how many C/C++ projects use Conan by default, versus those who don't?

For comparison all rust projects uses cargo.

The ergonomic value of that is something you simply can not overstate.

For many users of AOT compiled languages, the ergonomic factor is being able to enjoy fast builds, without getting a compiler rig.

Turbo Pascal 7, which was already relatively complex, was already compiling several thousand lines per minute on a single core 4Mhz computer bounded to 640KB.

More modern examples with complexer type systems, are languages like Delphi, FreePascal, Ada/SPARK, D, F# via .NET AOT, OCaml.

So yeah cargo is nice, but not if I have to build everything from scratch, Rust isn't a scripting language.

>So yeah cargo is nice, but not if I have to build everything from scratch, Rust isn't a scripting language.

FWIW, Rust builds faster than node applications as my laptop can only handle so many iops.

And Rust compiles are in line with Ocaml (probably faster by now due to the optimization work over the past year)

> But unlike C++, dependency management isn't hell

There is nothing hell into C++ if you are using the right tools: meaning proper package manager, meaning Nix/Guix, Conan or Spack.

It is currently much more hell to integrate Rust+Cargo behind other build system ( behind pip/gem/npm for instance ) than it is with C++.

I think you wanted to reply to the upper comment. :)

>Regarding Linux distros, if it isn't on the official repositories, Conan provides exactly the same experience as cargo, only faster because it supports binary libraries.

Don't use the operating system binaries unless you are packaging your application for use by that same repo. Instead, use Conan or you will find yourself in dependency hell trying to get users running it on Fedora, Ubuntu 12.04, 14.04, 16.04, 18.04, 20.04, Debian, etc where they all use different versions of the dependencies you want.

The official repositories are for the sysadmin and for other packages in the official repositories.

> As far as I can tell most of the people complaining about Rust having painfully long compile times are just parroting second hand information and don't actually know.

My experience is the opposite. Most rust devs don't work on very large projects from what I can tell, so compile times tend to stay manageable. It's a small percentage that work on projects large enough to hit this problem, and when you hit it it hurts.

We do have experience, my C++ projects compile much faster than their rewrite in Rust.

On common laptops that you buy at the shopping mall, not compiler rigs.

I am not doing nothing special, other than all my third party artifacts are binary dependencies, not overusing templates, incremental compilation and incremental linking.

Cargo currently doesn't do binary dependencies, so already here I have to wait for the world to compile.

And while incremental compiling is already supported, lld still isn't supported on stable.

Plus I do have experience using other AOT compiled languages with modules support, since back in the day.

So we know what are talking about, and plenty of people on Rust team are quite aware of the problem, however not every can be fixed at the same time.

> We do have experience, my C++ projects compile much faster than their rewrite in Rust

Wow. And C++ projects can be quite slow to compile

They can, but you have to compile everything from scratch, not use binary dependencies, disable pre-compiled headers, not having an incremental compiler and linker available, and be an heavy meta-programming user.

Reply on news.ycombinator.com