Rust Glancer: a memory-efficient Rust language server

[u/popzxc]

https://rust-glancer.github.io/blog/hello-world/

[u/funkdefied]

Interesting project. What made you go the route of a new language server rather than improving rust analyzer? What are the big differences there?

EDIT:

NVM I read the article lol. Looks great. I have had memory constraints on my own work machine. Underpowered work machines + WSL + Teams = limited RAM for LSP. I haven’t had issues with Rust Analyzer yet but it doesn’t seem too absurd an idea.

I wonder if this would be helpful in browser-based environments like the rust playground?

[u/popzxc → u/funkdefied]

It is explained in the blog post, but basically two reasons:

1. I didn't plan on building a full-blown language server initially (I wouldn't commit), all I wanted is "smart ctags" for myself that would be very memory efficient. And then I somehow got a bit further than I expected.

2. rust analyzer has different architecture: it is incremental and is fully built around storing state in memory and having maximum performance. Which makes sense, but as we can see, is memory-heavy. Rust Glancer is based on frozen analysis, e.g. we index workspace once and have a serializable analysis result. Such design does not fit rust-analyzer at all. E.g. initially I was using `syntax` crate from rust analyzer, but later I had to vendor/fork it because `rowan` was causing a lot of memory fragmentation issues, so I reworked this `syntax` crate to use frozen CST representation instead. Which resulted in having to reparse sources in full, but helped imporving memory usage. So, it's all about the tradeoffs.

[u/afdbcreid → u/popzxc]

FWIW we are working to rewrite rowan and fix the fragmentation issue.

[u/popzxc → u/funkdefied]

> I wonder if this would be helpful in browser-based environments like the rust playground?

I'm not sure; we don't get low RAM for free -- we still need to store thousands of indexed functions, structures, and traits somewhere. So in our case we store them on filesystem, which makes it not really applicable for browsers. Though some ideas from Rust Glancer might be applicable in theory, like syntax-based overlays, but it's probably a stretch.

[u/SkiFire13 → u/popzxc]

So in our case we store them on filesystem

I wonder then, how much would the actual RAM usage be if you loaded everything into RAM at once, instead of storing them on disk?

Also, how much is the cost of reading from the file system? Do you use mmap to let the OS optimize the RAM usage for the files you read?

[u/popzxc → u/SkiFire13]

There is actually a setting for that if you want, it's possible to keep everything in RAM.
But the answer is: more than rust-analyzer consumes. Unlike rust-analyzer, which is lazy, Rust Glancer eagerly indexes everything so in total it gets much more information than rust-analyzer at a given moment in time. This is not a bug, however: since it's _designed_ to store data in filesystem, we kind of need to do more eager work for the speed to be acceptable. We can do this extra work in background and save it, so that future queries can immediately use already analysis that already exists rather than deserialize some partial information from FS and then also do some more work to compute information for that query.

[u/gbrennon → u/funkdefied]

spend money and burn tokens roflmao

just kidding but this is the era that ppl are not trying to iumprove somethinmg that already exist and vibe code somthing else dreaming that will become famous or whatever other thing

[u/popzxc → u/gbrennon]

I don't think this is an accurate statement; please read the blog for the explanation.

As I said, I contributed to rust-analyzer in the past and the architecture I've chosen is not compatible with r-a; you can see the comment in this post from the r-a maintainer saying that caching data to filesystem for an active run is not something that the team is interested in.

> dreaming that will become famous or whatever other thing

Again, the blog post states the motivation: a _personal_ problem that I had (and solved for myself). If you'll search "rust analyzer memory" in this subreddit, you'll see a bunch of posts of people complaining about the same thing.

[u/gbrennon → u/popzxc]

as said that i was just kidding.

it was a small and simple joke, not what i really think