The Dark Secrets of Fast Compilation for Kotlin | Hacker News Reader

The Dark Secrets of Fast Compilation for Kotlin

blog.jetbrains.com

49 pointsingve18 comments

> (One could think of an even finer grained approach that would track changes in individual functions or classes and thus recompile even less than a file, but I’m not aware of practical implementations of such an approach in industrial languages, and altogether it doesn’t seem necessary.)

Lucid Energize did it: https://www.youtube.com/watch?v=pQQTScuApWk

Incomplete source code: https://archive.org/details/archiveteam-lucid-collection

Seems less “dark secrets”, more “incremental compilation 101”

It's a dark secret if you're used to maven and java...

Maven is incremental per default. Also java compilation is much faster than kotlin, so it is less of a problem.

Furthermore, IntelliJ (as much as I love it) has a terrible live compilation story compared to eclipse. Eclipse recompiles affected project files while you type and propagates errors live. Harp on java where it's at least factual, (crufty stdlib, missing uints, type erasure..). Java is miles ahead what tooling and maturity of the ecosystem is concerned.

Is it? Maybe our company is doing something stupid then.

I'm not mainly a Java person, but I do sometimes pitch in here and there. We're using maven, and it's definitely not doing incremental compilation. I mentioned this was crazy to the Java guys and they said it was normal for maven...

(They're also in an IDE most of the time, so maybe they don't care so much.)

They probably invoke maven with "mvn clean package" which does a fresh complete build, but only for deploying it.

I'm running "mvn install -DskipTests" w/ no files changed. It takes about 22 seconds and I see messages like

> Changes detected - recompiling the module!

> [INFO] Compiling 734 source files...

Perhaps we're doing something strange or dumb or somehow don't have incremental compilation turned on...

It seems[1] there's a flag called useIncrementalCompilation which is indeed enabled by default but doesn't actually cause any incremental compilation to happen.

Setting the flag to false causes it to compile only changed files, but this will give incorrect results, for the reasons described in the Kotlin blog post.

[1] https://stackoverflow.com/a/49700942

> Now, you have a basic idea of the challenges that fast compilation in a modern programming language poses. Note that some languages deliberately chose to make their compilers not that smart to avoid having to do all this.

I’m not knowledgeable enough to read between the lines, but is this a veiled reference to Go? If so, what about Go is intentionally not smart, and why? (Or language X)

The first sentence if the article: "Compiling a lot of code fast is a hard problem, especially when the compiler has to perform complex analyses such as overload resolution and type inference with generics."

The go language doesn't have either if those things, so the compiler has less work to do per line of code. But it seems like you knew that? You're the one that brought up Go.

There may be other languages that have taken a similar approach—I’m not up to speed on that—but simplifying elements of the language design to enable fast compilation was definitely a conscious design choice for Go.

See Rob Pike’s 2012 talk “Go at Google: Language Design in the Service of Software Engineering”

video + slides: https://www.infoq.com/presentations/Go-Google/

transcript: https://talks.golang.org/2012/splash.article#TOC_5.

Some of the things it calls out:

  - unused dependencies are a compile-time error

  - dependency graph has no cycles

  - importing a package only looks at object files, not source files, so template/macro stuff is generally out

  - declaration order is reverse from that of C so that parsing code is faster and easier

I felt maybe it was more taking a stab at Java? Since it was saying it wants type safety but without making the code ugly with type annotations everywhere.

I don't think they are referring to Go, a lot of languages avoid adding features that inevitably would add to compile times... Java for example (which I think is more likely what the author had in mind)... it requires you to end every statement with a semi-colon. Kotlin removes that restriction, but that costs compilation time (Kotlin seems to use a heuristic that if it's possible to terminate an expression when a new-line is reached without it being a compile error, then it does it... in some cases, a new-line is required even if it "looks like" it shouldn't be needed, e.g. `val func = {} val x = 0` doesn't compile). There are many features like this in Kotlin, and it shows as the Kotlin compiler is nowhere near the speed of the Java compiler if you don't use a freaking compiler daemon.

I had a test suite that used `StringSpec` to make tests prettier to write, but after a few hundreds of those, my Kotlin tests started taking visibly longer to compile than the Java source code (which was much larger)! So I had to re-write the tests to use plain Kotlin functions instead.

All languages I know that have "advanced features" have notably slow compilers: Rust, Scala, Haskell. I would be happy to know if there's any counter-examples (I might almost include Kotlin as a counter-example, actually, as it's not so bad if you let it run the Gradle daemon to use the techniques mentioned in the article - but that feels a little bit like cheating as you're getting fast by just avoiding compiling everything).

So while I think the trade-off is worth it most of the time, it's something to be aware of and consider in your decision when choosing a language.

Depends what you mean by advanced features, but OCaml has very fast compile times (usually faster than Go IIRC).

That would be truly impressive, given how OCaml has global type inference IIRC!

OCaml is usually compiled by module, and module-module interfaces mostly all have explicitly declared types (in the .mli), so it's usually not very global, just within the module.

Not many language have cross-module type inference. For example I'm pretty sure Haskell does not have cross-module type inference (you need `.hs-boot` files to help it along if you want cyclic modules for example). So in that sense OCaml's type inference is no weaker than other languages with global type inference.

And in fact type inference is generally not what slows down compilation. It is often barely any more expensive than type checking (and indeed for languages with global type inference a viable way to implement type checking is to run type inference and then compare the generated type signature with the annotated signature). This is especially true for OCaml which requires definitions to be in order or otherwise explicitly marked as recursive (so the type inferencing algorithm can immediately add a term's inferred type to the current typing judgment as a known type). What's slow is usually recursive implicit definitions and optimizations. This means typeclass-heavy signatures for Haskell and frequent use of recursive implicits in Scala.

One of the things go does is generate bad code [1].

[1]: https://lemire.me/blog/2020/06/04/the-go-compiler-needs-to-b...

Reply on news.ycombinator.com