Rust Analyzer in 2018 and 2019 | Hacker News Reader

Rust Analyzer in 2018 and 2019

ferrous-systems.com

181 pointsjamesmunns50 comments

Hey! This is a post by matklad [1], one of the members of the Ferrous Systems [2] team. The Rust Analyzer project [3] is an experimental compiler front-end, designed to power things like code completion, syntax error reporting, and other things that are necessary when building IDE support. Let us know if you have any questions!

[1]: https://github.com/matklad

[2]: https://ferrous-systems.com

[3]: https://github.com/rust-analyzer/rust-analyzer

Sure. Great work! Would rust-analyzer provide enough of a foundation to build something like this?: https://gtoolkit.com

Mind you, in the IDE above reflection builtin the language allows for the incredible amount of power. In Rust's case, the only way I can I see it replicated is at a layer before compile time, hence my interest in your analyzer.

Unlikely: rust-analyzer focuses on static analysis. So it aims to power something akin to IntelliJ IDEA, but probably would be a bad fit for something like lightable/elisp/smalltalk ide.

To better understand why this project is trying to re-implement the compiler frontend, I really recommend this video from Anders Hejlsberg (creator of C# and TypeScript) explaining how modern compiler construction oriented towards IDE works:

https://www.youtube.com/watch?v=wSdV1M7n4gQ

There is another approach to bootstrap an incremental compiler frontend on top of a batch compiler, which I used to create a Java language server [1]. Batch compilers usually support incremental compilation at the file level, so you are already close to a solution: re-compile the active file every time the user edits. This works but is too slow. You can dramatically improve the speed by simply erasing all the code that isn’t relevant to the current edits, which usually means erasing all the method bodies. This simple trick allows you to build a fast incremental compiler on an existing batch compiler.

[1] https://github.com/georgewfraser/java-language-server

This sounds like a use for the everybody-loops compiler flag: https://github.com/rust-lang/rust/pull/19964

We (Sourcegraph) have also found the “dump all the data from the compiler” approach limiting because it doesn’t work for the live editing use case (as the article says), and then that means you have 2 analyzers: one used with editors and one used for offline tools like Sourcegraph, DXR, etc. That bifurcation means each one gets less community involvement. The ideal IMO is to always target the live editing use case and add caching and optimizations that use language-specific knowledge, so that in both cases you’re querying a live server but it can compute the answers quickly. I haven’t worked with Rust specifically, but it’s interesting to see that they came to a similar conclusion.

BTW Sourcegraph is interested in funding this effort and building better Rust support for code browsing and search into our product using https://docs.sourcegraph.com/extensions. I just reached out to the author.

You guys created language server protocol, right? I think langserver.org is owned by sg?

No, Microsoft created LSP, but we’ve built a lot of language servers and added LSP for code intelligence in Sourcegraph (and on code hosts like GitHub and GitLab with https://docs.sourcegraph.com/integration/browser_extension).

(Aside: are you from n-gate.com, as your username suggests?)

Oh right. I just googled. Why add two canonical sources when the original github does a fine job and is also publicly editable (by pull requests)?

Microsoft told us they definitely did not plan to make any kind of language server registry, so we saw a community need for an informational site to exist with more details than the wiki page they had. We’ve heard it’s useful, but we don’t want to create duplication. Want to file an issue on the langserver.org repo for discussion?

Love your about page!

You are violating The Prime Directive.

What would that be?

If you can't figure it out, perhaps you are truly a Hackernews. :-)

I don't see why one should be well versed in stupid hollywood-isms.:)And hackernews is a derogatory term that I will have you know I am very offended by. Better only than web-boy

The thing is, if you target only live editing case, you can become slower for "batch compilation" case. For source browsers this probably isn't too important, b/c one can add caching using something like LSIF. However for command line compilers a similar caching won't work: in the majority of cases, you compile new code. But adding fast "batch mode" on top of "live" editing seems easier than going in the opposite direction.

How often does one do non-incremental compiles though?

I think it's pretty common for CI, but I guess that use case can handle a little lower performance.

And for each CI compile, how many times have devs done an incremental compile? Probably a lot.

I've worked on projects that took over an hour to compile. I would not like lower performance for CI. Many workflows require successful CI runs before one can merge for example.

It's really only Rust and C++ that ever take that long, right? IMO, that's simply the compiler being unacceptably slow. I also can't see a slightly less efficient parse tree being a big factor in compile times this kind of length...

Well with a good build cache, CI can do things like incremental compiles too. You can even put the cache on the network and have most of your CI use incremental compiles.

Eclipse does make this work for java with ECJ. It gains total and correct knowledge of the code base by instrumenting a compiler designed for incremental compilation. But it's also usable for batch compilation.

There's a whole class of problems around this that I'd like to see get some more attention. It shows up for all kinds of static analysis in code editors, including things as simple as autocomplete.

For instance, if I have a file that compiles and I start typing in the space between two function bodies, I'm probably implementing a new function. In fact that probability is so high that you should just assume that to be the case, even if the code I've written so far doesn't have balanced nesting yet. All decisions and advice should be based on the speculation that the code will eventually represent a new set of blocks, not a reorganization of the remaining ones.

You can't make the same assumption about if statements inside of a function, or about incomplete multiline comments, but you can assume that anywhere from zero to all of the sibling blocks are about to have their scope changed (but not ancestors).

In short, the gap between how I'd describe a commit and how the diff tool represents it also prevents live code analysis from being able to work as well as it might.

It seems like parsing both forwards in the normal way and backwards from the end of the file would be helpful? That is, take the longest suffix that parses correctly and assume any syntax errors are before that for error-reporting purposes.

Most programming languages aren't designed for parsing backwards, though!

Backward in time, not a linear file scan.

That's not a particularly hard problem, all tools which try to solve it (for example, IntelliJ), solve it easily. For example, here's the syntax tree that rust-analyzer builds for this example:

    fn foo() { }

    fn

    fn bar() { }

Syntax tree (whitespace nodes elided for clarity):

    SOURCE_FILE@[0; 31)
      FN_DEF@[0; 12)
        FN_KW@[0; 2)
        NAME@[3; 6)
          IDENT@[3; 6) "foo"
        ...
      FN_DEF@[14; 16)
        FN_KW@[14; 16)
        err: `expected a name`
      FN_DEF@[18; 30)
        FN_KW@[18; 20)
        NAME@[21; 24)
          IDENT@[21; 24) "bar"
        ...

The problem is, a lot of tools are build with the assumption that the code is correct&complete, and don't try to tackle this problem at all.

When llvm was young, one of its design goals was to be useful for live code analysis, to prevent that problem of having two parsers that might not agree.

Sounds like we lost some of that along the way.

It would be interesting to develop Xi and rust-analyzer in tandem, and see how much of an IDE Xi could become while leaving it still basically a text editor.

Language features are definitely in the near-term plans for xi, and depending on what stage Rust Analyzer is at it could definitely be a good initial target for some of that work.

Does xi have any Vim/modal like features?

We're still trying to figure out the best way to implement those. There's a prototype, but it hasn't landed in master. It's definitely on the roadmap though.

I thought Kakoune was an interesting experiment: an attempt to embrace vi-like modal editing, but keep it consistent, and go all-in on it.

I really want Kakoune to succeed. I love the UX behind it and it feels like a vim-done-right.

The major innovation of kakoune, imo, is to combine modal editing with multiple selections. These are two very powerful concepts that haven't been combined well before, to the best of my knowledge. In any case yes, I think kak is a great editor and a definite source of inspiration.

Sounds cool! I don't know what to say about Vim-like. At this point I think it's mostly a dream that one day, something can replace Vim. Not saying it's perfect, it's just so entrenched, and full of features.

That said, I don't think I'd be after a perfect Vim copy. I think I would like an editor that uses the same approach, doesn't have to be exactly the same keybindings.

I agree this could be very interesting.

I find building solid foundation and tooling really important, and it seems where are right there at this point with Rust. But IDEs have always been controversial beasts. Some might say Visual Studio at one point was one of the best representations of good set of trade-offs that made great code-writing environment. Now it's different story. But what I am far more interested in in terms of Rust and it's IDE ergonomics is what can be achieved differently? Rust by itself is different beast. Unique language that pushed different way of thinking and structuring code. So can that be used as an inspiration for something different in terms of IDEs and developer UIs?

I am not sure I understand the criticism of RLS. It feels to me that the Language Server Protocol does allow for incremental parsing etc. In particular, the approach taken by rust-analyzer

> Analyzer maintains a "database" of input facts (source code + information about project structure) and derived facts (syntax trees, name resolution information, types). The client then can change input facts, and query derived facts at the current state of the world.

seems that it could also be adopted by a language sever for Rust.

The criticism are not about the protocol itself, but about a particular implementation. rust-analyzer also uses LSP as an interface.

The criticism is not directed at the LSP. RLS just implements the specification, but as you said LSP doesn't create these constraints.

One outcome of all this might be that RLS internally gets replaced by something like rust-analyzer, which is something that the maintainer of RLS has mused in a recent blog post as one of many possible options for the future of RLS:

https://www.ncameron.org/blog/more-on-rls-version-numbering/

>The only fundamental problem with IntelliJ Rust is that it is not written in Rust, and I would really love to have a pure-rust solution for IDEs. Why? The main reason is that Rust is the perfect language for building these sorts of tools

Citation needed.

I assume you are talking about the last bit? This is of course a personal opinion anyone is free to disagree with. However in practice many popular programming languages are implemented in C or C++ (clang, gcc, swift, CPython, V8, SpiderMonkey, HotSpot) and in theory the best language for compilers is [Oca]ml. Rust is a nice synthesis of theory and practice.

That is mostly by convenience than anything else, as many devs don't want to bother with the whole process of bootstrapping a programming language.

Hotspot long term roadmap is to be rewritten in Java for example, aka Project Metropolis. And not all JVMs are implemented in C or C++.

I think it's pretty clear from context that this is the author's opinion.

As an end user, one thing I care a lot about is if the dev env would be truly integrated. I want a one-click/command install of everything that I need for my vim to be a rust IDE. I tried in vscode and had to install two different plugins (one of which had a similar name to another and I had to go look up which one to install) python and then lldb using admin permissions which failed on my school computers .

Side: Has emacs been losing traction slowly? It's a third of vim!

AFAIK it was always behind VIM. People learned vim because of its ergonomics and install base.

Also many people did not know TRAMP is a thing so yeah...

Wasn't emacs farther ahead for full-time programmers though? I'd love to see if there were some historical stats.

For me XEmacs was the go-to environment while I didn't had access to Borland like IDEs on UNIX environments.

Nowadays XEmacs seems to have died, I have plenty of IDEs to choose from for my favourite languages, even on GNU/Linux and Emacs seems to still lack some of the nice features of XEmacs.

As for VIM, I learned VI on Xenix, but never could bother to actually use it more than for profile files, or as the go-to editor over telnet/ssh sessions when nothing else was installed by default.

I assume many on the IDE camp might share a similar experience.

Reply on news.ycombinator.com