Hacker News Reader

Comment by potamic | Hacker News Reader

Could you elaborate a bit on why RA's incremental approach takes more memory? Intuitively it feels that it should take less, because you're only processing what you need? Whereas you seem to indicate that you save a full analysis snapshot to disk and load it all up when needed? Shouldn't that consume the max memory for a workspace?

Fantastic project btw, and it couldn't come at a better time. With the way prices are going I really hope people start paying attention to memory again.

View on HN

It's explained in the blog post, but in short: rust-analyzer stores the data it needs in memory all the time, while Rust Glancer might consume more memory during indexing (because it's not lazy and does more indexing), but after that it only loads _necessary_ information for the duration of the query.

Several things here: 1. We don't need all the information (project can have 1000+ dependencies, while query might only care about the current open file), so the amount of information we load is smaller. 2. Most of the time IDE does not actually do any queries, so if you switch to browser/Slack, you don't pay the tax. 3. Since data is loaded to the disk, after initial indexing restarting no longer consumes that much ram, and you get reindexing for free. 4. Besides offloading, I implement quite a bit of memory optimizations (some of which are covered in docs: https://rust-glancer.github.io/docs/development/MEMORY.html ), so it's a combination of factors.

Let's say I type "SomeStructure" and haven't imported it, how does rust glancer _not_ end up poking at all the dependencies to figure out where it is, and to import it?

In some sense I feel like that example is the canonical "you need a full picture of the world" case, to the point that it's probably worth special casing _just_ that, and using local analysis for everything... but it seems hard to avoid that pain on any typo for example

Inversely... do you have a ballpark for what the size of the on-disk structures end up being? Not that it matters _that_ much but Rust projects already have a tendancy to be disk hogs during development

On the first question -- auto imports is an example of a "heavy" functionality since we indeed need to scan more than is imported in the project, which is why I'm still working on this to optimize properly (it's decently fast, but I want it to be faster before I'll enable it, since it's also something we want to see in completions, and completions must be _very_ fast). Still, we don't need "everything": we only need _reachable_ items in _direct_ dependencies; typically project has much less direct dependencies than transitive ones (e.g. 20 direct dependencies can fan out to 1000 transitive deps), and each dependency usually has less exported items than total amount of items. Finally, for that you only need semantic data (e.g. items) rather than bodies. So while it's a big task (though finding references is still significantly bigger and tougher, because that's where it gets to "it can be anywhere in reverse dependencies bodies), it's not "we need the whole world".

As for the size -- the Rust Glancer artifacts for Rust Glancer itself currently take 225mb. And that's proportial to the project size only, e.g. it doesn't grow over time on its own.

Unfortunately that does not work. Consider:

    // Crate indirect_dep:
    pub struct Foo;
    
    // Crate direct_dep:
    extern crate indirect_dep;
    pub use indirect_dep::*;
    
    // Crate my_crate:
    use direct_dep::$0;

($0 denotes the cursor). You cannot know what to complete without analyzing `indirect_dep`. Add macros to the mix and you're going to get a nightmare (yes, name resolution in Rust is a nightmare).

In fact, you sometimes need to do type inference inside bodies to infer other bodies, because auto traits propagate across opaques. You might say this does not matter because it only matters for diagnostics... But it does not. It can matter for method resolution.

In general, I'm fairly sure that it is just impossible to build a 100% correct analyzer for Rust code without a query system like rustc's (it can be stored on disk, that's a different story). You can go pretty far without - and this will be enough for some people for an IDE! - but not all the way.

Reply on news.ycombinator.com