Hacker News Reader

Comment by sqs | Hacker News Reader

We (Sourcegraph) have also found the “dump all the data from the compiler” approach limiting because it doesn’t work for the live editing use case (as the article says), and then that means you have 2 analyzers: one used with editors and one used for offline tools like Sourcegraph, DXR, etc. That bifurcation means each one gets less community involvement. The ideal IMO is to always target the live editing use case and add caching and optimizations that use language-specific knowledge, so that in both cases you’re querying a live server but it can compute the answers quickly. I haven’t worked with Rust specifically, but it’s interesting to see that they came to a similar conclusion.

BTW Sourcegraph is interested in funding this effort and building better Rust support for code browsing and search into our product using https://docs.sourcegraph.com/extensions. I just reached out to the author.

View on HN

The thing is, if you target only live editing case, you can become slower for "batch compilation" case. For source browsers this probably isn't too important, b/c one can add caching using something like LSIF. However for command line compilers a similar caching won't work: in the majority of cases, you compile new code. But adding fast "batch mode" on top of "live" editing seems easier than going in the opposite direction.

How often does one do non-incremental compiles though?

I think it's pretty common for CI, but I guess that use case can handle a little lower performance.

And for each CI compile, how many times have devs done an incremental compile? Probably a lot.

I've worked on projects that took over an hour to compile. I would not like lower performance for CI. Many workflows require successful CI runs before one can merge for example.

It's really only Rust and C++ that ever take that long, right? IMO, that's simply the compiler being unacceptably slow. I also can't see a slightly less efficient parse tree being a big factor in compile times this kind of length...

Well with a good build cache, CI can do things like incremental compiles too. You can even put the cache on the network and have most of your CI use incremental compiles.

Eclipse does make this work for java with ECJ. It gains total and correct knowledge of the code base by instrumenting a compiler designed for incremental compilation. But it's also usable for batch compilation.

There's a whole class of problems around this that I'd like to see get some more attention. It shows up for all kinds of static analysis in code editors, including things as simple as autocomplete.

For instance, if I have a file that compiles and I start typing in the space between two function bodies, I'm probably implementing a new function. In fact that probability is so high that you should just assume that to be the case, even if the code I've written so far doesn't have balanced nesting yet. All decisions and advice should be based on the speculation that the code will eventually represent a new set of blocks, not a reorganization of the remaining ones.

You can't make the same assumption about if statements inside of a function, or about incomplete multiline comments, but you can assume that anywhere from zero to all of the sibling blocks are about to have their scope changed (but not ancestors).

In short, the gap between how I'd describe a commit and how the diff tool represents it also prevents live code analysis from being able to work as well as it might.

It seems like parsing both forwards in the normal way and backwards from the end of the file would be helpful? That is, take the longest suffix that parses correctly and assume any syntax errors are before that for error-reporting purposes.

Most programming languages aren't designed for parsing backwards, though!

Backward in time, not a linear file scan.

That's not a particularly hard problem, all tools which try to solve it (for example, IntelliJ), solve it easily. For example, here's the syntax tree that rust-analyzer builds for this example:

    fn foo() { }

    fn

    fn bar() { }

Syntax tree (whitespace nodes elided for clarity):

    SOURCE_FILE@[0; 31)
      FN_DEF@[0; 12)
        FN_KW@[0; 2)
        NAME@[3; 6)
          IDENT@[3; 6) "foo"
        ...
      FN_DEF@[14; 16)
        FN_KW@[14; 16)
        err: `expected a name`
      FN_DEF@[18; 30)
        FN_KW@[18; 20)
        NAME@[21; 24)
          IDENT@[21; 24) "bar"
        ...

The problem is, a lot of tools are build with the assumption that the code is correct&complete, and don't try to tackle this problem at all.

When llvm was young, one of its design goals was to be useful for live code analysis, to prevent that problem of having two parsers that might not agree.

Sounds like we lost some of that along the way.

You guys created language server protocol, right? I think langserver.org is owned by sg?

No, Microsoft created LSP, but we’ve built a lot of language servers and added LSP for code intelligence in Sourcegraph (and on code hosts like GitHub and GitLab with https://docs.sourcegraph.com/integration/browser_extension).

(Aside: are you from n-gate.com, as your username suggests?)

Oh right. I just googled. Why add two canonical sources when the original github does a fine job and is also publicly editable (by pull requests)?

Microsoft told us they definitely did not plan to make any kind of language server registry, so we saw a community need for an informational site to exist with more details than the wiki page they had. We’ve heard it’s useful, but we don’t want to create duplication. Want to file an issue on the langserver.org repo for discussion?

Love your about page!

You are violating The Prime Directive.

What would that be?

If you can't figure it out, perhaps you are truly a Hackernews. :-)

I don't see why one should be well versed in stupid hollywood-isms.:)And hackernews is a derogatory term that I will have you know I am very offended by. Better only than web-boy

Reply on news.ycombinator.com