Comment by matklad | Hacker News Reader

Comment by matklad | Hacker News Reader

Parent Full thread

Descriptively, yes, the shape of the language determines how you can structure an interactive compiler. There are at least three wildly different ways to do that:

https://rust-analyzer.github.io/blog/2020/07/20/three-archit...

Which one can work depends on the language in question. Java, Rust, C++ lead to different answers, and that’s not because of the “interesting” differences like borrowchecker vs gc, but rather due to the “boring” differences in name resolution and module system.

Prescriptively, whether we _should_ do this (or, rather, how much), is unclear.

My biased answer is that, while language designers talk a lot about making languages tooling friendly, there’s little of that actually happening (at least with “current” languages, “next” ones seem to fare better). Like, it was said that “rust macros are designed with tooling friendliness in mind”, but overall the language is pretty tooling-hostile, mostly for accidental reasons. It seems to me that if we _actually_ co-design a tooling-first language, without trying to innovate too much, but by just ensuring that existing techniques work robustly, we might arrive at a close, but meaningfully different point in the design space.

Specifically, here’s my version of IDE-friendliness diff for language design:

Push conditional compilation far further in the pipeline, such that it is done after all semantic analysis. This is a pre-requisite for making automated refactors which are _guaranteed_ to work.

Similarly, push meta programming further down, such that code analysis doesn’t invoke user-defined code (which might be arbitrary slow), but, at the same time, meta parts can fully reflect on existing code, including resolved types. C#-style source generators are an interesting design here.

Have strong signatures on micro and macro level. Macro, have well-defined compilation units with explicitly specified dag of dependencies and signature files. Micro, annotate types of the functions. Use tooling to reduce double-annotation burden.

Ensure that each source file can be somewhat deeply analyzed in complete isolation. The last two points should unlock both embarrassingly parallel (distributed) compilation, and snappy completion.

> there’s little of that actually happening (at least with “current” languages, “next” ones seem to fare better).

I'm not too involved but I think the languages that actually develop the tooling as part of the language design have traditionally done quite well at it in language design. Delphi/C#/TypeScript all had tooling built alongside the language and it shows.

Rust sadly made a ton of choices in the language that just make a lot of things hard, but enabled the type of ecosystem that exists today. So it has different tradeoffs. I'm quite amazed how well RLS is doing considering how hostile the language is to fast iteration.

I am not entirely sure about C#, if we speak about “static analysis based IDE” tooling. I think this only became focus with Roslyn, and Roslyn came later, after JetBrains did their own thing with resharper.

But yeah, directionally we are getting there: Microsoft seems to be the main popularizer of the idea (TypeScript and LSP), and Google seems to be most acutely aware about the “tooling gap” (Go, Dart, and, most recently, Carbon are cognizant of tooling needs. Carbon has signatures for tooling, and Dart shipped a better version of LSP before LSP became a thing).

I just remember years ago reading interviews with Hejlsberg about how he designed C# (particularly LINQ) with IDEs in mind. Given that he also worked on Delphi I think it’s a core pillar.

I think that was meant in a very narrow sense comparing to SQL. In SQL the columns being selected go before the the table name so the IDE can't suggest a set of valid columns. In LINQ the order is flipped. But other than that there's no grand language design there to support tooling.

You don’t have to have a grand design, you just need to get the important pieces right. C# has an excellent IDE experience not because LINQ is in the right order but because it has a fundamental compilation model and object system that supports tooling well. They could have gone nuts with compile time logic but they did not to support a better IDE experience.

Look at Kotlin to see the opposite. It was designed to be IDE friendly - in fact, it was developed by the IntelliJ folks.

However, looking at it now, it seems that Kotlin is pretty much falling behind even Java in terms of features and functionality. But especially it also seems behind in a well-thought design.

When looking at Rust, I think while there are still a lot of features that I would desire, it at least introduced major improvements like typeclasses (or "traits" in Rust lingo) which really help the ecosystem and libraries to flourish. If this makes it harder for the compiler and the IDE then I think it's a fair tradeoff. Kotlin had the chance to do the same, they deliberately didn't and I is already starting to hinder the language from thriving.

Curious as to what parts of Kotlin you think are falling behind Java. Java has been playing catch-up forever. Java just implemented light weight threads - coroutines in Kotlin which I've been using for years. I've used data classes forever and, again, Java just recently implemented their equivalent - records. Sealed classes are the only feature I can think of but not a must-have for me. What am I missing? For some background, I've been a Java developer for over 10 years. I work with Kotlin and Java daily - my company has a backend written in Kotlin and many ancillary services in Java. I prefer Kotlin - it's a joy to write in. JetBrains hit a sweet spot by being able to mix functional and imperative coding.

A good example is the latest pattern-matching addendums to Java which are lacking in Kotlin in comparison. Sealed classes (aka coproducts/sumtypes) are another and are an essential part of any programming language, on the same level es enums. I think you are experiencing the blub-paradox in this case.

Yes, Java has been (and is still) playing catch-up, but there has been some serious progress recently and when features come, they tend to be better than the Kotlin counterparts from what I can tell.

That being said, I'm not claiming that Java is better than Kotlin right now. But when projecting into the future, I doubt that Kotlin has one, sorry to say that. A couple of years ago I would have said the exact opposite, but Java has really stepped up since then.

Wait, if sealed classes are essential, how did java survive so long without them? Possibly a nice-to-have but essential? To say Kotlin doesn't have a future when it is one of the widest used languages is a stretch. I think it's great that Java has been stepping up their feature development. I will continue to use both.

Well, java "survived" without it because, at the time, most other languages didn't have sumtypes either. But almost every new/modern language does. Exceptions are rare (golang is one of them and even for golang there are discussions: https://github.com/golang/go/issues/57644)

> To say Kotlin doesn't have a future when it is one of the widest used languages is a stretch

Well, let me say it like that: if Java and Kotlin continue to progress with the same speed like they do now, then Kotlin will not be a "better Java" anymore and the only selling-point will be IDE support (if that's even possible).

Kotlin is a 'better Java' because of the ability to mix functional and imperative code. Java will never do that. I don't know what you mean about 'IDE support'. Neither of them are going anywhere.

Kotlin is made by Jetbrains which make IntelliJ - so IDE support is a plus-point for Kotlin. That is what I meant.

I think Java has become way more functional over time. It is not and probably never will properly support a pure functional style (e.g. like Scala) but I don't see it much behind compared to Kotlin. Mind to give some examples where Java is behind Kotlin in this context?

Kotlin is in a tricky place w.r.t. Java because the JVM is one of Kotlin's target platforms. If Kotlin deviated from the JVM too far, it would be harder to migrate to the newer JVM versions.

That specifically applies to things like pattern matching and value classes. Value classes are constrained to single values to allow the Kotlin compiler to do the heavy lifting without involving specific JVM capabilities.

It will be interesting to see how Kotlin evolves with version 2.0 and beyond, with the new compiler and stable multiplatform support.

I don't think so. Look at Scala, which is in the same position as Kotlin. Still, it has a far superior featureset both compared to Java and Kotlin and has had pattern matching since almost forever (in fact, Java's pattern matching, sealed classes etc. are heavily inspired by Scala's). And it manages quite fine to do all these things, even before the JVM improved to support those features better in a native way.

Thanks for that.

It seems that some newer languages (like in the last ten years) either care about (1) batch compilation speed or (2) language design mostly unconstrained from having to have fast batch compilation. It would be interesting to see a third way: language design that takes into account all kinds of future interactive tooling by laying a good groundwork. So maybe from-scratch batch compilation ends up slow, but then it turns out to be fine in practice since you can distribute builds and cache things so that you only pay at most a one-time large cost. And then what you mostly have to deal with is incremental stuff.

Interactive proof languages already work that way, to a greater or lesser extent.

But, for general-purpose languages, this revictimizes the docker addicts.

It’s hard to push down conditional compilation too far without compromising its value, but one can still stop at a much later point than Zig does. (In fact, the fundamental choices of Zig in regards to compile time necessarily makes it hard to create good IDEa for)

In my C-like (C3), I retained an #if-like construct, which while semantic (unlike C’s) still meant dealing with sections of conditional code that had to be resolved before semantic checking could commence. To simplify I removed the ability to conditionally include struct members etc.

Doing more research I found D’s `version` interesting. In particular the non-block variant where the version is attached to a declaration.

I ended up creating a more flexible (and therefore actually less good) version of it as an attribute (`@if(cond)`) which then may be attached to any declaration. The advantage here is that at most this leads to some symbols having two (or more) different declarations (resolvable using the `@if`), which allows a lot of analysis even if the attribute hasn’t had its argument resolved.

I think this might be the way forward at the top level.

Inside of functions / macros, $if and friends are fine as they often do not matter for the overall analysis.

Alex, I've been reading all your posts on Zig and Rust LSPs with great fervor and appreciate everything you've done for the community with rust-analyzer. Can I ask you about clangd, the llvm/C++ lsp? It seems to work tremendously well, far better than I naively ever thought it would even on massive codebases. I was using it from its first release and the quality ramped up rather quickly. Can I bug you to write up about it from your perspective and with your insights sometime and where we could benefit from their design choices and where we can't? And of course, the limitations of their approach?

It's ostensibly driven by the `compile_commands.json` produced by the compiler during a run (mapping each source file to the output .o and the command (w/ all options) that was used to generate it) but in reality that's just because C++ doesn't have its own build system so the tooling has no other way of knowing how a project is wired up. But aside from that, it's extremely fast and has fairly low latency for a language that's got a lot of the same warts that rust has (generics, monomorphization, macros, strong types, (limited) type inference, etc) especially when compared to all the other LSP offerings (although I think C++ compilation units are smaller than rust's, which has to help).

I covered this a bit here:

https://rust-analyzer.github.io/blog/2020/07/20/three-archit...

I don't actually _know_ how exactly clangd works, especially post modules, but pre-modules C++ has a compilation model which is actually quite friendly for an IDE, because of the header files.

Headers essentially explicitly encode body/interface separation that more advanced incremental compilation systems like Salsa try to recover implicitly.

Looking at a slightly simplified typical C++

    #include <iostream>

    void main() {
        std::cout << "Hello, World!" << std::
    }

what a language server could do is that it can run bog-standard phased compiler, and just _freeze_ its state after all includes are parsed&expanded. After that, any typing inside the file needs to re-analyzer just the file itself. Also, the code in included `.h` files is typically far smaller then the code in the corresponding `.cpp` files. So, a language server gets ability to analyse only the relevant code, and skip the rest, for free, it is encoded in compilation model.

Compare this with Rust, where, in the general case, changing a file can affect any other file in CU in arbitrary ways (_usually_ it doesn't, but it _could_, and the complexity lies precisely in figuring out which edits are local and which are not). And CU's themselves are meatier, as macro expansion can run arbitrary code, so you need to somehow cache that, taking into account that macro expansion _anywhere_ in CU can potentially affect _anything_ in CU. So, you need full build systems a-la carte incremental compilation, which is a) hard b) different from how the command line compiler works.

The second point explains the success of clangd I think: _usually_, when you have a command-line compiler and want a language server, you want to rewrite, because the architectures are too different. But because of the way C++ (and OCaml) work, for those two languages you actually can soundly re-use existing code with relatively minor modifications.

> Push conditional compilation far further in the pipeline, such that it is done after all semantic analysis

So much this. I've idea how this would work un practice (like how you're supposed to accomodate antagonistic semantics which depend on the conditionnal compilation), but I hate #[feature(X)] with passion because of how brittle they are.

Reply on news.ycombinator.com