Crazy fast build times, or when 10 seconds starts to make you nervous (2012) | Hacker News Reader

Crazy fast build times, or when 10 seconds starts to make you nervous (2012)

dan.bodar.com

108 pointskiyanwang137 comments

My personal theory of modern software development is that the #1 and #2 things you can to do improve software quality are:

1) Make sure devs can run a realistic version of the whole stack locally.

2) Drive incremental build times and effort on the whole stack to zero.

With today's world of frameworks, dependencies, and microservices it's too hard to just "think through" a change. You need to try it, adjust, and try it again. Reducing friction on the iteration cycle makes this task pleasant and increases the likelihood of discovering the best solution. When this cycle is painful devs are likely to stop at the first working solution (or even partially working).

I find 1) especially challenging with microservices. What technologies do you use/recommend to help with this?

Docker compose and very beefy dev machines.

This is where a Docker/K8s containerised setup can help.

Not trivial to get to, but for deve/test environments it can be really beneficial. Can also mean the same clean starting point for every series of tests, if new containers are spun up each time through.

how do you combine this approach with some external services (i.e. databases aren't run in docker but AWS RDS or some part of the setup is using AWS lambdas)?

Even if you have your prod database on RDS, you might be able to spin up a local instance for testing. My Postgres-using applications have a script that sets up and runs a local Postgres instance for the duration of the test. It takes about 2 seconds on first run, and less than noticeable time when the DB has already been initialized.

https://github.com/sapcc/keppel/blob/master/testing/with-pos...

Is the RDS Postgres identical to whatever you have installed in a docker? How would you do this with Aurora?

> Is the RDS Postgres identical to whatever you have installed in a docker?

I don't use AWS myself, so not sure. But if it's not the same, doesn't that destroy their value proposition?

not necessarily. If the positives of the RDS environment are more important to me than running my own database on EC2 in a docker with the latest features, i am totally fine with giving up some DB related features.

We have been using RDS for production and normal Postgres during development for about 5 years and the only thing you have to watch out for we found is that RDS supports only certain extensions and they are fairly slow making new Postgres major versions available. So e.g. if you want to use TimescaleDB you either have to run it yourself or use a proprietary AWS alternative.

It is just a Postgres running, but you have limited control about its configuration.

So as long as you make sure during development that you are not doing something with the database that RDS doesn't support you should be fine.

I don't know about Aurora, but I reckon it is similar. Standard Postgres features are likely going to work fine, but more advanced things are not going to be available on Aurora.

Basically Postgres in Docker would be highly flexible in terms of what versions (major or minor), extensions, your own customisations for the configurations in pg.conf you want to go with.

You can upgrade/patch as soon as Postgres community makes these available. With RDS you have to wait and for Aurora surely significant waiting periods! e.g. Aurora Postgres is currently 2 major and 2 minor versions behind - meaning you would be missing the latest and greatest of Postgres!

I would suggest to use Postgres from community on EC2 or Containers or if you want Postgres as a service then use Postgres operator on EKS or any K8S distro available on AWS. You can easily go back and forth between this env and RDS or even any platform making your Postgres databases completely platform agnostic.

https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...

Not to mention all of the EKS/GKE integrations that pretty much everyone on the platform uses. Things like EFS backed persistent volumes, ALB backed ingress, ELB backed service load balancers, etc. You need some way to abstract over the different stacks because you can’t just deploy your YAML to any Kubernetes distribution and have it work properly.

https://github.com/localstack/localstack

Is a pretty good testing framework that we use at my workplace for a dockerized microservice setup. Requires some config but has mocks for most of the big services (specifically RDS looks to be in their paid tier, but we get along w/ the free version just fine.)

Not that many companies actually need microservices. Consider going back to a more monolithic setup

I'd be happy if the developers were given an "average" device for testing, and not the newest-shiniest-flagship one.

I get it why for development it's beneficial for a test device to be as fast as possible (because there less time is wasted for (eg.) an app to start after a change in code), but a lot of modern applications don't do more than they did 10 years ago, but eat up a lot more resources for the same output.

I find this is true for phones also (or even moreso). Google maps is a bad offender, getting slower every year despite doing essentially the same thing.

Google (allegedly, in public) does this, although for some NIH-smelling reason they've invented their own representative low-end phone instead of just buying some.

I'd like it if they tested their distributable product on a hard drive sometimes, not just SSD all the time.

I've come across a game that had major stutter problems, very noticeable on a HDD but still noticeable on SSD, just noticeable on a ramdisk, problem was frequent saving level data to disk and having some things block while they did that. The developer got lots of reports about performance, they said that people should upgrade their computer, game was 2d and graphically simple, the developer did not understand why people were having performance problems and didn't really care, it is essentially abandoned at this point and still for sale.

Another game also had major stutter problems which would last a few seconds each, turns out it was reading a file (which should be read once at startup) over 100 times a second almost all the time it is open. Submitted bug reports in both places they support with log output from the file monitoring I did. No response to either of them.

Firefox generally runs fine, if the HDD where your profile is located is saturated with IO then pages will not load until Firefox is able to write to the disk, this is with disk cache turned off. This isn't unexpected behaviour and not a bug like I would consider the previous two examples, but it is noticeable.

I really would like developers to test on a HDD even if only every few releases, and would also like them to use a monitoring tool on their software sometimes. This is just so their products don't have really bad behaviour on the system it is running on which the developers are completely oblivious to.

I've been intentionally developing a data science type app - one where run times are critical - on a 12' MacBook.

I know run times will feel great on a 6 Core i7 machine for my fairly simple test cases. With the MacBook I get immediate feedback if something I've done creates a slowdown.

I kept my 10 yo laptop only for testing my apps on regular consumer hardware, not my typical $3k overkill dev machine.

Another reason why you want a blazing fast test device is that running with certain debugging and profiling tools attached can degrade performance substantially.

I definitely agree things should be tested on realistic hardware - but one of the big reasons for me at least is using a decked out machine is because of all the dev tools + browser tabs, etc. IntelliJ might be using 6GB, chrome another 6GB, 6GB for teams/discord/figma/gitkraken/vscode/spotify. That's not including any image editing software or vm's but I'm already needing over 16GB of ram.

Exist an incredible good reason: Because MOST of the tools we must/need to use have atrocious performance.

Not bad. Terrible.

I use Rust, and I like as language. But is terrible experience for compile times. I know what a good one is (Delphi, Pascal) and after it no single compile time language get it close (with exception of GO).

The terrible thing is that the base of all is C/C++/JS and their toolchain is the SLOW incarnate. Then even if I dodge the use of them, I still get impacted, eventually.

For example, I use TailwindCSS... that is great. But it use node. Node is terrible. I must use the speedier machine I can pay because soon or later I will deal with node, c++/llvm/c, Android Studio (damm!) Android emulator (mega damm!) and other tools like that.

And the tricky part ("why not just not use them?") is that they are terrible at performance but maybe good for something else (or: I must use them... I'm contractor!).

So only IF I have total control and can supply my needs with better tools I could use a old machine of the past. But I can't.

It would be awesome if Rust were faster, of course. At least with Rust, you get the borrow checker, not just compiling. It feels like you're getting something more in return (even though I don't know which step of compilation actually takes the most time).

But that would still suck. Luckily, there is a solution that works in many cases, rust-analyser. Obviously having basic syntax checking in an editor is preferable, because it shortens the feedback loop. I think this is why a lot of IDEs do it, not just for Rust. It's still a work-around, but makes it more bearable.

I'm not hugely experienced in Rust development, but it would be awesome if cargo could compile files when they are saved, so incremental build times and running tests might feel shorter. You could still get into trouble with complex dependencies between code units, but better > perfect. Visual Studio's "Live Unit Testing" takes this to an extreme (yeah, VS itself has poor performance, but doesn't mean the idea is bad).

Basically, if tools must have bad performance, because they are doing a lot of work for us (?), then at least make the development experience not suffer from that bad performance. That's what I care about more than just perf, it's how it impacts me, right?

In general, I still think compiling provides a decent trade-off. Yes, I need a beefier machine, and yes, my dev cycle is just a bit longer. But people using the actual product, they get the benefits. You could go with e.g. Python, where maybe one or two tests take less time to run, but ultimately you'd just be shifting the burden somewhere else. It isn't much consolation in the moment, but worth remembering.

One thing I saw was someone logging all system calls their software generated and comparing that between runs. They'd been bitten by some minor library upgrade that had changed directory scanning which caused huge multiples of normal latency which on an SDD wasn't noticeable but on an HDD went from trivial to insane.

That was handy because it didn't actually involve testing on a slow device but mostly showed what we needed to know.

There are tools like https://github.com/schoentoon/slowpokefs which try to make your fast storage act like a HDD which could be better for black-box testing.

It would be nice if it was easier to have a setup which is fast for development but realistically slow for testing.

I develop my personal website on a 6-year-old Windows laptop which was only $750 at the time. It definitely keeps me from making a slow website! But it also makes my devserver much slower than I'd like. I wish I could have speedy tools but a slow browser (and no, Chrome's CPU throttling isn't really the same as a truly crappy computer)

At least as important for software quality is:

3) Make extension of thorough automatic testing in all levels of the testing hierarchy as effortless as possible.

Nobody is going to bother with thorough testing if it takes a day to add a simple test case for the new minor thing they just added in the last half an hour. Boom, test hole, disaster waiting to happen next time somebody refactors something. No matter how great you are doing with 1) and 2).

> 1) Make sure devs can run a realistic version of the whole stack locally.

This idea is greatly under-appreciated in the industry.

I'd love to be able to do this where I work, but we have a configuration of our software requires a minimum of 16 machines to work properly. And that is full blown machines, not just tiny dockers or whatever, but our AWS bill is in the 10s of thousands just for our 1 test environment for that setup. That test environment is undersized massively from the production environment, so we can't even do real world load tests against it. It doesn't have the disaster recovery enabled (which would double its size and adds a f5 big-ip load/proxy server in front.

We also have a configuration that is more like a single server mode, which is what we primarily develop against on our local setups, but it isn't the exact same functionally, and there are always defects when deploying to the other configuration.

I’ve been working on native (Swift) components within a React Native app, and the single-line edit/rebuild time was about a minute, which was just killing me.

Creating a separate Xcode project which includes only the Swift code dropped the edit/rebuild cycle down to about 6 seconds.

This is definitely an issue with microservices, but how do frameworks contribute? I've moved more toward monolithic frameworks over the last year (Next.js and Phoenix specifically), and it feels to me like local setup is easier and build times are shorter than when I'm rolling more of my own setup.

I agree frameworks help with the dev setup through all the cool hot-reloading tricks, I just meant because of their complexity it's one more thing you have to keep running in testing. I don't want to go back to jQuery in a <script> tag but that had 0 startup cost.

I recently was asked what I liked about common lisp, and one thing I struggled to put into words was how good the #2 experience is.

SBCL is an implementation that is AOT compiled to machine code, yet the incremental build time is shorter than even python because you can recompile a single (file|function|system) without restarting your program. Python fights you if you even want to re-load a single imported module (IIRC there are ways to do it, but it doesn't seem to be a typical way of developing in python).

> Make sure devs can run a realistic version of the whole stack locally.

And through the magic of docker-compose, they can! Now just let me wait while 33 local containers spin up so I can unit test this one-line fix...

As a summary, switched from scala → java, ant → make, removed GWT, Lucene, Hibernate, Velocity, Tomcat etc.

This needs a (2012) on it, but it looks like that got hugged to death already [1].

[1] - https://web.archive.org/web/20210324055256/http://dan.bodar....

When I was still deep in the Java space, I always used hot code replace, otherwise it’s really unbearable. I also contributed a patch to hibernate to cache the compiled classes. That sometimes saved minutes per restart.

It's important to note that Scala compile time got better, both with improvements in the compiler and in the computers we're using (going from HDD to SSD was huge, and of course CPU improvements).

Still slower than Java but I think it's well worth it for the productivity gain.

> Still slower than Java but I think it's well worth it for the productivity gain.

I have never seen this play out it practise. The most productive engineers I know can touch type well and know their editor/IDE inside-out, have a solid grasp of the basics and preferring not too many abstraction layers, and a workflow optimised for fast dev cycles. Sure, build your startup with Scala or Haskell - I'll bet on the Ruby / Golang / Lisp competitors.

My project has 17k lines of Scala code (no whitespace). I don't have a SSD. It takes 10 minutes to perform SBT Assembly from a cold start, 40 seconds afterwards. And about 1-3 seconds to compile and run locally within IDEA.

Kotlin compile times are killing me.

In a current project, I have fewer than 20K lines and it's 50ish seconds for a clean re-compile on a very decent laptop (less than a year old, 32Gb RAM, quadcore i7, fast PCIe/NVMe SSD, etc.).

Particularly as I'm a test driven type of developer - cycling between running unit tests and adding small fixes is painful. Even when you can trust the incremental compilation, 10 or 20 seconds between making small one-line fixes and the test re-starting means the temptation to check hackernews or whatever kicks in and any chance of achieving "flow" evaporates.

I'm almost considering the suggestion (for Scala) in the article - and moving back to Java.

it's a nightmare isn't it. Are you using the 1.4 compiler?

Yes it's a nightmare. I'm using 1.4.30.

In response to your question, I just bumped it to 1.4.32 and haven't observed a significant improvement.

Back in the old days, compiler metrics were often quoted using units of 100s of thousands or even millions of lines per second.

I know the Kotlin compiler has to do a lot of work but I would have imagined hardware advances should balance this out somewhat. Instead I'm looking at compile speeds quoted in hundreds of lines per second.

And I was complaining about a 4s rebuild time for a similar (number of lines) .NET 5 solution

does it not have incremental compiles?

at least Scala incremental compiles are fast. Haskell too!

Compile time is the only reason i moved to reactnative. It has some issues, but its well worth to have the fast build cycle.

The Kotlin compiler is slow, but if you break up your code into smallish modules and build with Gradle, it will only re-compile the modules that require re-compilation after changes which should make compilation times much shorter than 50 seconds.

At work, we have around 100 modules I think (some Kotlin, most Java though) and changing the API of the lowest level ones (used by everything else) usually causes re-compilation to take a minute, but as most changes are in the higher level modules (which are not depended on by anything else) and re-compiling those takes a second or two, it's not been a big problem for us.

I envy your 50ish seconds. Especially since it sounds like you could cut that down quite a bit with a beefier build rig.

I do kernel development, and for me the turn around for a one-line change is 4-minutes to 2+ hours. About 60 seconds for the build (32-core Threadripper, solid state storage), about 30 seconds to tar up the kernel and scp it to a test box, and about 2 1/2 more minutes to install the kernel and reboot (most of which is in the super slow uEFI bios). That's the 4 minutes for a trivial change.

Somewhere in the middle is ~40-ish minutes for our central CI service to run all the unit tests when a branch is pushed to.

But the 2+ hours is what I encounter most often. That is for testing non-trivial changes on production traffic. It takes over an hour to gently ramp a box down from full load to no traffic (so I can install a kernel an reboot), and another 1+ hours to ramp it up with customer traffic. Since we're testing in production, it means I often want to watch the machine closely, so that I can gracefully ramp it down if things start going pear shaped.

I agree that leaves way too much time for distraction. I try to multi-task and have multiple projects in flight at the same time in order to make more effective use of the waiting time.

I see your 2+ hours and I raise you 1/2 of a server year of test regressions for every commit to the main trunk, beginning with a 2 hour wallclock-time build of all the models needed for running CPU simulations. Total wall-clock time for edits to hit the trunk is 2.5 days.

Android development is the worst. Not single thing about it is good. All slow, all the time.

And Gradle, what kind of sick joke is that?

BTW: Is possible to replace Gradle for android development with something good???

I use hot-reloading of tiny modules to get turnaround below 1 second on all my projects, server and client:

My Java HTTP server hot-deploys a .jar containing new code with a simple classloader: https://github.com/tinspin/rupy

My C+ 3D MMO engine uses .dll/.so hot-reloading so that the game code can change while the engine is running, it's not released yet: http://talk.binarytask.com/task?id=5959519327505901449

Very nice work. About hot-loading: Where do you put the frontier between each modules? How big is each of them? Maybe you could share a typical module list?

Keep the good work!

This is one of the things that really makes me hesitate with Rust.

I have a pretty small database project which sometimes needs to slowly stream updates to a web browser. Pulling in actix and actix-web added over 1s to my debug mode build time on a fancy new zen3 computer. The web front end isn’t the core of this project - which makes the slowdown really sting. I’m tempted move all the web / actix stuff behind a compilation flag just to keep my coding flow while working on the rest of the project.

I know compilation time is being worked on, but it’s still way slower than it could be. I’d love to see an incremental linker for rust some day that’s able to just replace the methods in the binary which were actually changed between builds.

I agree. Hopefully when the Cranelift backend works it will get a lot faster, but even `cargo check` can be pretty slow, and I'm not sure if there is any way around it.

One idea I had: can crates.io generate machine independent Rust IR for all of the crates and have cargo download it? Then at least you don't need to parse/check/compile the source code of dependencies for clean builds.

Probably wouldn't work for any crates that use `build.rs` or conditional compilation anywhere though (which is probably most of them).

Apart from the conditional compilation problems, you would also have to pre-compile for all different Rust versions. Given the size of a typical `target` directory the artifacts for just a single Rust version will also probably be ~10x as big as the current crate source.

Mmm good point. An ideal solution for CI would be something like sccache, but I think to get that to work reliably you need to consider it from the beginning (like Bazel did) and Rust just didn't. `build.rs` can do literally anything. Proc macros can do anything. Even a simple `include_str!()` can break distributed compilation.

It would be great if Cargo had a "hermetic" mode where you couldn't use `build.rs` and proc macros couldn't do any IO, etc.

It's an interesting idea, and one python has sort of used for a while with wheels (pre-compiled binary packages for specific architecture/OS combinations). For Rust, they could potentially cache the MIR or even a higher level LLVM IR, but perhaps those formats are too unstable or end up extremely complicated.

as long as you are not switching compilers during development, those formats are stable..

Is it mostly compilation time, or link time? At least for debug builds, I found swapping out the linker to work wonders. There are some examples on https://endler.dev/2020/rust-compile-times/ (no affiliation, just 1st google result)

Disabling debug information also helps, regardless of the linker.

Enabling opt-level=1 also does wonders.

It enables dead code elimination, and removes a lot of code very early.

Awesome thanks for the tips everyone! I wasn’t expecting that. The slowdown is almost all linking time. I’ll try swapping out the linker and playing with build settings.

Currently use Rust, was anxious about build times but incremental builds on my machine done with 'watch' are no concern.

Switched to a faster linker though.

Also 'rust check' is what I most often need which is fast for me. But perceptions are subjective.

Using Scala for a long time, Rust compile times are a relief. Not as fast as Go and not as fast as Typescript (which I used with live compiling and Wallaby, Rust is quite away from TS unit testing turn around).

I know actix uses lots of generics and macros to match up your functions with varying signatures to Handlers for a request. That might be a reason why it’s slow but I don’t know enough about Rust compilation to be sure.

Can’t you put the web stuff in another crate that depends on the main library crate? That way you can build and test your library without all the actix/web stuff.

Just use warp. Warp is a lot faster to compile but is also less performant. If web isn't a core part, it's not worth the incremental speedup.

Ok I'll take a look at Warp if I can't get my compile times down with the suggestions from others.

I initially wrote my code using Tide, but I'm streaming data through long-lived HTTP responses (with a protocol similar to SSR). Tide made this extremely difficult to pull off. Actix (not actix-web) makes this much cleaner, because I can create an actor per web client. The actor owns the connection, and I can send messages to the actor to be translated into bytes and sent over the wire. Even if I use something other than actix-web, I might keep using actix for the actor based programming model (which is a good fit for my project). And if I'm doing that, actix-web seems like the obvious choice for the web component.

I have a 5900x and an SSD.

For example project like fluent-rs takes about 300-700ms to compile?

So I don't know what kind of projects are you running when you hesitate to use Rust?

I have a small project that has around 2k lines now, and it takes over 5 seconds to re-compile. Fast compilers can do 2k lines in less than a second. waiting over 5 seconds for unit tests to run can actually be a problem as it slows down your cycle.

Sure but fluent-rs is not a small project. I don't have exact numbers but definitely above 2k lines. Even if you exclude the comments.

What I mean by it, maybe it's not the line count, but using too much macros and/or monomorphization.

An incidental aside, but I actually wrote a comment yesterday on the benefit of using the fastest tooling you can find, along with a restart-from-scratch strategy (non-incremental, non-"reload these specific things while juggling ABI compatibility with X Y and Z already in memory") similar to clean rebuilding.

It's a textbook wall of text, but IMO represents a bit of anecdata that 100% agrees with the idea that lowering build and processing times are absolutely worth it. https://news.ycombinator.com/item?id=26555366

Is anyone else (who is not using a caching DNS server from their ISP or other publicly available one) having trouble resolving the domain? There is no A record -- in fact there is no domain at all as far as the root servers are concerned: http://www.dollardns.net/cgi-bin/dnscrawler/index.pl?server=...

Snippet from the above link (to keep this comment relevant once things eventually get fixed):

  Querying a.root-servers.net (198.41.0.4)... delegated
    Querying e.gtld-servers.net (192.12.94.30)... name not found
    Unable to find: dan.bodar.com

    No Answer Records

Looks like the domain record was recently updated to clientHold (about an hour after this article was posted):

  server ~ # whois bodar.com
     Domain Name: BODAR.COM
     Registry Domain ID: 4584805_DOMAIN_COM-VRSN
     Registrar WHOIS Server: whois.joker.com
     Registrar URL: http://www.joker.com
     Updated Date: 2021-03-24T07:06:55Z
     Creation Date: 1999-03-22T05:00:00Z
     Registry Expiry Date: 2022-03-22T04:00:00Z
     Registrar: CSL Computer Service Langenbach GmbH d/b/a joker.com
     Registrar IANA ID: 113
     Registrar Abuse Contact Email: abuse@joker.com
     Registrar Abuse Contact Phone: +49.21186767447
     Domain Status: clientHold https://icann.org/epp#clientHold
     Name Server: IVAN.NS.CLOUDFLARE.COM
     Name Server: JEAN.NS.CLOUDFLARE.COM
     DNSSEC: unsigned
     URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/

clientHold apparently means[0]:

> This status code tells your domain's registry to not activate your domain in the DNS and as a consequence, it will not resolve. It is an uncommon status that is usually enacted during legal disputes, non-payment, or when your domain is subject to deletion. Often, this status indicates an issue with your domain that needs resolution. If so, you should contact your registrar to resolve the issue. If your domain does not have any issues, but you need it to resolve, you must first contact your registrar and request that they remove this status code.

It's quite surprising how many people on HN are talking about this article just because of however long the TTL was on the original DNS query and somehow almost everyone so far has got a cache hit.

[0] https://icann.org/epp#clientHold

Presumably they got hugged to death and are currently trying to change to a different server or something. I can't access it either.

From the definition of clientHold, I don't think it's related to any traffic increase. I don't think the domain owner can request that status, nor could I think of any reason why they would want to!

When I query it:

  dig @8.8.8.8 A dan.bodar.com

I sometimes get an A record and sometimes no record. Repeating this query @1.1.1.1 never gives the A record.

Anyways here is a mirror / archive link for anybody trying to read the article: https://archive.is/2RUaI

As it's real close to the anniversary of registration, I wonder if the domain holder renewed the domain late, and that's settling.

Back when I compiled on a cray I had problems with make because files could compile within a second so make would want to recompile them.

I use a lot of C++ templating so these days fast builds are just a fantasy no matter what I do. Unlike the author I rely on incremental builds for development but do a full build before pushing

c++ templates are a real killer.

Have u tried a cloud compiler? Did your company ever try a build server ?

I work in video games, large AAA game with absolutely crazy amount of templating, the compilation time is about 10-15 minutes, and that's with Fastbuild and hundreds of workstations participating in the build process. Once it's built it's a bit quicker, about 60 seconds from pressing build to the game starting, the linking is killer, it takes soo long every time.

> the linking is killer, it takes soo long every time.

did you try using lld for linking ? even on windows, it's much much much faster than link.exe

Yes, we use Clang for everything, including lld

No use of external templates or having common subset in a binary library and then use private inheritance?

There are a lot of techniques to help, but they only help so much.

I could opt for less metasyntactic power but it would make the code less expressive, harder to follow, and more error prone. The tradeoff is worth it.

Given I used to compile on a .25 MIPS KA-10 there's little to complain about. For a while in the mid 80s I had two symbolics lispm consoles in my office because I would use one as a build machine and one for development and other stuff (one had a gasp color display attached as well!). That was when programmer time was worth the capital cost. Honestly it didn't matter as much as incremental development was done with interpreted code anyway.

But time spent compiling is time spent reflecting, and doesn't have to cost you flow.

And at the other end of the spectrum you have 1M lines of code compiled in just 19 seconds, using latest Delphi compiler (https://blogs.embarcadero.com/delphi-10-4-2-compiler-speed-i...)

Great news... Compile time or a clean build? I use similar tooling (older version... :-)). And I'm starting to evaluate what's going on with those new versions.

If you watch the video closely you'll see he's doing a full build for the entire components using a test project.

Pascal was an awesome learning language alone for its compile speed.

It still is. Under Linux you have Free Pascal + Lazarus and can target, natively, everything under the moon (see target section for: https://en.wikipedia.org/wiki/Free_Pascal)

Somehow webpack/JS/React has way faster build times than Java/Android apps. That's the main reason we completely abandoned fully native apps in Java/Swift in favor of webapps with a tiny native wrapper app for the few only-native calls we need.

And to move forward this trend, new bundlers are appearing in JS like esbuild/snowpack that are even faster than webpack.

This is why the web will always win in the long run. While the Android team in working on their tooling and Apple on other tooling and Windows on other tooling on the web all tooling is shared. You don't need the OS/platform gatekeeper to come up with faster bundlers. You just need one company/opensource team to make a breakthrough tool.

Finally this is the point why I believe TYPES ARE NOT WORTH IT. Types make the whole development slower. To me is better to be able to try and run the code faster in development and try it manually several times than to wait longer for type checks. Typing is overrated in my opinion.

With esbuild, typed code compiles just as fast as untyped code (esbuild doesn’t typecheck just strips the types) so I don’t buy your last argument. Esbuild gives the best of both worlds, imho.

The problem with esbuild is that it doesn’t do hot reload (yet?)

I don’t like hotreload, so I don’t see it as a problem. I do a full rebuild and my browser auto refreshes and that’s good enough for me. I’ve had hot reload get me into an invalid state too often to find it useful.

For working on complex webapps that take a bunch of steps to arrive at a certain state, hot reloading can save a ton of time, since you don't have to redo the steps to get there.

One of the bigger time-savers for web development, in my opinion.

I've never tried esbuild. I've only saw the benchmarks.

Frankly everytime they say this new X language/tool manages types fast it turns out not to be true. Deno has native types and then it came out that the actual Deno source code decided to abandon Typescript: https://startfunction.com/deno-will-stop-using-typescript/

https://esbuild.github.io/content-types/#typescript It seems esbuild has a separate loader for typescript and anyway it only builds but does not type checking: "However, esbuild does not do any type checking so you will still need to run tsc -noEmit in parallel with esbuild to check types. This is not something esbuild does itself."

> even faster than webpack

Even faster than webpack? This thing is dog slow to me. Sit here compiling for 40 seconds every time I restart my app.

That may be nice if you previously had minutes of compile time, but I came from PHP where everything was instant.

> Typing is overrated in my opinion.

For very small projects, maybe. For large 100k+ LOC projects, most definitely not.

Well I was comparing client-side apps like Android/Java VS React/JS.

PHP vs React/JS is not really an apple to apple comparison because you still need client code to achieve high interactivity like Gmail did first.

Few questions on how to improve your webpack build/hot-reloading time: Are you on webpack5? Are you using Yarn 2 PnP? Did you break the app in multiple import() modules?

we have a very large mono-repo all with like 30 services 10 webapps and 4 native app wrappers. The main B2B app is the biggest with like 50 separate pages and 100 modals/dialogs. The build time is around 40 seconds to produce the production bundles. More impotantly than that though is the weback serve/dev-server/hot-reload update time and that takes 1-3 seconds. Essentially if I can a letter in the UI and I save it take 1-3 seconds to see in the browser. This is with the largest webapp we have. All other webapps refresh in less than a second and essentially time you switch from the editor to the browser the page has already been refreshed.

Webpack 5 for example is significantly slower than Webpack4. There are hacks to speed it up but you need to make workarounds or turn off features to get acceptable speeds. This makes it needlessly complicated to configure and manage project just to compile JS...

>Somehow webpack/JS/React has way faster build times than Java/Android apps. That's the main reason we completely abandoned fully native apps in Java/Swift in favor of webapps with a tiny native wrapper app for the few only-native calls we need.

I'm sure your users are glad you're giving them a worse experience because your build times are marginally faster.

If you want crazy fast build times, then D is your best bet [1]. D's compile times are faster than most other languages (I'm not counting Vox, it's too experimental).

[1] https://github.com/nordlow/compiler-benchmark

Wow you're not joking... `dmd` is around 40x faster than `rustc` and 16x faster than `zig`! How the hell they've done it?

Check Zig again once the self-hosted compiler is completed :)

Will that remove the dependency on LLVM?

I definitely agree with "no Fitnesse", sorry, FitNesse, ugh, and likewise, no Selenium.

Not so much for the slowness. In the case of FitNesse (ugh), it's the Wiki -> Java impedance mismatch, and in terms of Selenium, it's the brittleness. Move an element, watch Selenium tests generated by a QA's browser plugin break.

I rather loathe Hibernate these days (JDBI is my favourite atm), and well, Tomcat feels rather legacy. But I guess that's the advantage of 9 years of advances.

> you can use the maven structure and produce maven artifacts with your build but use a different tool (Make, sbt, buildr, ant etc)

Omg, no, sbt is far, far, far, worse. Ditto ant. Not sure if buildr is still a thing? Looks like the last release was 2017. And as for make, how would you even integrate various Maven plugins into it?

> JDBI is my favourite atm

Tried JOOQ? There's generator overhead on db schema changes, and it adds lots of classes to your classpath (but this could be inside a precompiled library). I guess it adds some build time, not sure how much though (for sure more than JDBI). But that sweet typesafety and auto-complete is worth something.

Interesting choice of JDBI. I was working on an SQL-friendly ORM[0] also due to distaste with Hibernate/JPQL and chose JDBI, not because it was great in any way but it did what I needed and not much else. What influenced your choice and were there any close runner-ups?

[0] https://github.com/karmakaze/safeql

have you looked into ktorm?

I wasn't looking at Kotlin libraries, but this one seems to be very close to what I was going for. Specifically about being able to compose queries in a bottom-up fashion like the UnionAll example with subqueries.

It's unfortunate that they chose to use .filter (.where would have been clearer) for DB executed conditions as well as post-fetch sequence filtering.

For us, it was basically about using direct SQL to avoid the ORM / SQL mismatch, while still having a (relatively) straightforward way to turn rows into POJOs.

I mean, not as easy as just annotating some attributes of a class for Hibernate, but being able to specify the SQL itself was what sold me on it.

I guess it's not much different to Spring's JdbcTemplate etc. in that regard, but it came with minimal dependency baggage.

I do quite like Jdbi's SqlObject API, wherein you annotate a interface's methods with SQL (and a bunch of other annotations if you're getting fancy) which abstracts the boilerplate away to a large extent. I use this quite often for integration testing where there's an embedded PG instance and you need a straightforward way to populate it with expected data.

Where it really shined though, was in stuff that is trivial in SQL, but hard in DSLs, like doing a batch upsert (or as PG calls it, INSERT ... ON CONFLICT DO UPDATE...) was trivial in Jdbi compared to Hibernate etc. Just because we weren't relying on some DSL to generate the necessary SQL.

Also, Jdbi integrates reasonably nicely with Flyway schema migrations.

Thanks for sharing that. I completely agree with your assessments. My motivation was similar, mainly fighting with Hibernate/JPQL and wanting something simpler that was structured the natural bottom-up way an SQL query would be written. Another thing that ORMs don't do well is avoiding N+1 queries when you're already starting with N of something that have associations. I started SafeQL to see where it could go but have yet to do any code generation which is what it needs to be usably convenient. Also haven't ventured much into complex UPDATEs or UPSERTs.

That was an interesting read for someone like me who just started splitting builds into frontend/backend to get build times down. That they would switch languages just to get faster builds sounds extreme to me. But in the end I already noticed how much better it feels when the CI-cycle is minutes and not tens of minutes.

The suggestion to split off code with long-running tests into separate libraries is easy pickings. So I think I'll do that next time I wait for those tests to complete.

I'm procrastinating on HN right now because my iOS build time is 2 and a half minutes long. About 30-50 seconds on an incremental build. This is true suffering.

Oh wanted to update with Specs; 2019 MacBook Pro with 2.4Ghz Intel Core i9, 32GB RAM.

Are you using CocoaPods, SwiftFormat, or SwiftLint?

Running SwiftLint or SwiftFormat in a build phase killed my compile times (SwiftLint especially). It'll help a lot to run them manually or with a git pre-commit hook.

-----

I know Swift had really bad compile times for awhile, but anymore I think it's pretty fast. I've moved all my WIP stuff into a single project (113 Swift files, ~5,500 LOC) for my convenience, and incremental builds are around 3 seconds on a 2015 MBPr.

This article shows some flags you can use to find code that is slow for the compiler to deal with: https://www.avanderlee.com/optimization/analysing-build-perf...

Right now I can't stand that my Create React App builds for a TypeScript project take several minutes to build on CI machines. Makes me want to ditch it entirely!

Do it. My Preact builds with esbuild take a hundred milliseconds or thereabouts. The slowest part of my build process is waiting for TwilwindCSS to minify.

rollcall of essential tools if you are in C/C++ land:

no build should be without https://ccache.dev/, and after that, distcc https://distcc.github.io/, icecream https://github.com/icecc/icecream, and goma https://chromium.googlesource.com/infra/goma/ may be used for distributing the workload.

Can I use that with GitHub actions? I have some crazy slow C/C++ build times with O2 and asan, which usually leads to g++ timing out. Not just 40min build times. Some huge autogenerated testcases to cross-test all iterator combinations against C. I'd really need -O3 for -finline, but -O2, -O3 and -Ofast all fail too build. Splitting it up didn't help. So far I went with #pragma optimize O1. https://github.com/rurban/ctl/actions/workflows/master.yml

C++ templates are the worst. The pure C variant builds in seconds. I pity all C++ folks

my problem right now is that the OOM killer, is, err, killing compiler processes when asan is enabled. A build machine should have at least twice as much RAM in GB as it has number of cores for compiler processes.

Wayback machine if the site is still down for anyone:

https://web.archive.org/web/20210324071034/http://dan.bodar....

Comparing compile times is done rarely. I sometimes tried to google some comparisons, but didn't find much.

For a typical 1,5 MLOC project my clean build times are sitting around 40 seconds.

I think that's pretty good, but I don't think I have a way to improve it much. Without writing a compiler from scratch I think 10 seconds are out of reach for me...

Do u think your company will consider cloud compiler if this is x5 faster? There issues such as latency, delta synchronizing, and tool versioning. But its doable.

If u build such a service (pun intended!), the issues are probably cost and security.

Improvements in build-time are very valuable to me. But I'm not sure if a cloud would help me much. Synching can easily eat up 10s. Also my tools are very unique (including a parser that I wrote myself).

To be faster than local a cloud would have to be massively overclocked - and I can already reach good results locally.

For actual improvements I need a massive improvement in compiler technology (last time I looked there were just too many single-threaded bottlenecks in my build process). Nothing a cloud can solve for me.

Security is one issue - but hackers will most likely only get confused when they try to understand what is going on - also costs I usually don't like...

The thing that worries me about this post is that they are making a lot of sacrifices in the name of build speed. They are switching languages and libraries and changing their code. Basically, they are creating some amount of tech debt in the name of build speed.

Maybe because I started out with C++ on a relatively slow computer, build speeds have not been much of an issue for my productivity. Especially with code completion/etc running and underlining in red my errors as I typed.

I guess I kind of treated build as an asynchronous process where I could spend time thinking about the next thing I wanted to do.

That being said, definitely get an SSD and a beefy machine, as low build times are nice, but not worth changing your language, libraries, and code structure.

Apache Maven can be fast and easily sub 10 second if you just use in a straight-forward manner.

From an arbitrary mid-sized project: About 400 classes split into 10 submodules. Clean build without tests: 3.7 seconds.

Introduce spring and friends and prepare to see that number jump up at least 4-5x

Did he measure how much worse the Java version is to work on? Scala infers a lot of boilerplate and wasted effort a lot faster than I can write or review it, because I'm made out of meat.

I've started experimenting with the Bazel-like https://please.build/ , which has actual support for incremental builds.

I also advocate avoiding a lot of the typical dependencies in the Java ecosystem. There is a lot of bloated garbage that you'll miss less than you might think.

What are the typical dependencies, in your mind?

I should say lots of typical dependencies are fine, but some are common "problem children" or ways to shoot yourself in the foot.

The Spring ecosystem comes to mind. With regard to the development cycle, Spring seems to turn things that normally would be compile-time errors into runtime errors. If I've wired up my components incorrectly, I'd prefer to find out sooner rather than later, so as not to waste my time.

Ah yep, thought you might have been referring to Spring, and totally agree on runtime vs. compile time.

I've switched to Micronaut of late for that precise reason - all the injection happens at compile time, so far quicker to pick up.

Downsides though are that Micronaut sometimes doesn't provide super-helpful error messages - or silently fails to do DI when `micronaut-inject-java` isn't in the dependencies.

QCon talk (2012): https://www.infoq.com/presentations/Crazy-Fast-Build-Times-o...

Interesting. I've always been aggressive on trying to keep my Java and lately Kotlin builds fast. Anything over a few minutes in CI becomes a drain on the team. Basically a productive team will have many open pull requests open at any point and lots of commits happening on all of them. That means builds start piling up. People start hopping between tasks (or procrastinating) while builds are happening. Cheap laptops become a drain on developer productivity. Etc. All of this is bad. Maintain the flow and keep things as fast as you can. It's worth investing time in.

Some of the overhead is unavoidable unfortunately. E.g. the Kotlin compiler is a bit of a slouch despite some improvements recently. Many integration tests these days involve using docker or docker compose. Overall that's better than a lot of fakes and imperfect substitutes. But it sucks up time. A lot of Kotlin and Spring projects involve code generation. This adds to your build times. Breaking builds up into modules increases build times as well but tends to be needed. Be mindful of all this.

A few performance tips not covered in the article that may also apply to other languages:

- Run your tests concurrently and write your tests such that you can do so. Running thousands of tests sequentially is stupid. When using junit 5, you need to set junit.jupiter.execution.parallel.enabled=true in platform.properties (goes in your test resources). Use more threads (e.g. junit.jupiter.execution.parallel.config.dynamic.factor=4) than CPUs for this as your tests will likely be IO limited and not CPU limited. If you are not maxing out all your cores, throw more threads at it because you can go faster. If your tests don't pass when running in parallel, fix it. Yes, this is hard but it will make your tests better.

- Don't do expensive cleanup and setup in between tests. This takes time and integration tests become more realistic if they don't operate in a vacuum (your production system is not a vacuum either). To enable this, randomize test data so that the same tests can run multiple times even if data already exists in your database. Docker will take care of cleaning up ephemeral data after your build. This also helps with running tests concurrently.

- Distinguish between (proper) unit tests and scenario driven integration tests as the two ideal forms of a test. Anything in between is going to be slow and imperfect in terms of what it does. This means you can improve test coverage (of code, functionality, and edge cases) by making it a proper integration test or faster by making it a proper unit test (runs in milliseconds because there is no expensive setup).

- With integration tests, add to your scenarios to make the most of your sunken cost (time to set up the scenario). Ensure they touch as much of your system as they can to do this. You are looking for e.g. feature interaction bugs, heisen-bugs related to concurrency, weird things that only happen in the real world. So make it as real as you can get away with. A unit test is not going to catch any of these things. That's why they are called integration tests. Make it as real as you can.

- Fix flaky tests. This usually means understanding why they are flaky and addressing that. If that's technical debt in your production code, that's a good thing. Flaky tests tend to be slow and waste a lot of time.

- Separate your unit and integration tests and make your builds fail fast. Compile + unit tests should be under a minute tops. So, if somebody messed up, you'll know in a minute after the commit is pushed to CI.

- Eliminate sleep calls in tests. This is an anti pattern that indicates either flaky tests or naive strategies for dealing with testing asynchronous code (usually both). It's a mistake every time and it makes your tests slow. The solution is polling and ensuring that each test only takes as much time as it strictly needs to.

- Run with more threads than your system can handle to flush out flaky tests. Interesting failures happen when your system is under load. Things time out, get blocked, deadlocked, etc. You want to learn about why this happens. Fix the tests until the tests pass reliably with way more threads than CPUs. Then back it down until you hit the optimum test performance. You'll have rock solid test that run as fast as they can.

- Keep your build tools up to date and learn how to use them. Most good build tools work on performance issues all the time because it's important. I use gradle currently and the difference between now and even two years ago is quite substantial. Even good old maven got better over time.

- Pay for faster CI machines. Every second counts. If your laptop builds faster than CI, fix it. There's no excuse for that. I once quadrupled our CI performance by simply switching from Travis CI to AWS code build with a proper instance type. 20 minutes to 5 minutes. Exact same build. And it removed the limits on concurrent builds as well. Massive performance boost and a rounding error on our IT cost.

Most of this advice should work for any language. Life is too short for waiting for builds to happen.

I like https://github.com/awaitility/awaitility for testing asynchronous code.

It's funny how hardware is SO FAST today but yet we can't get the performance people used to get from... Borland Turbo Pascal in the 80s!

back in the HDD day, plenty of memory + a ram disk would really help.

I recently experimented with a ram disk. Practically it didn't change anything.

OS-caching seems to already be clever enough and once the OS has figured out that some directories are important anything in there seemed to get done in RAM anyway.

A RAM-disk will make this less black box and more deterministic regarding guaranteed access times, but in daily use the RAM-disk just didn't make a difference.

Which OS? And were you already on SSD? Those can both greatly influence the outcome.

I'm on Windows. Other OS may differ. I experimented a lot with load times - and compared HDD, SSD, RAM-Disk.

I wrote a small program that loads everything from a big code-repository into RAM. The first time HDD and SSD and RAM-Disk make a big difference, when reading files a second time the lag of HDD (50s?) almost disappeared completely. Caching kicked in.

The RAM-Disk has less initial lag, but also it has to be filled first, so instead of moving everything to RAM-Disk just touching everything so the OS-Caching kicks in is just faster and more convenient.

When I used to use Go, I always wondered if there were stale cached object files because the tests and builds were so fast. I'm not complaining, because I want nearly instant builds or even continuously running background live builds.

Reply on news.ycombinator.com