Does Lemmy benefit from Rust? Is code execution speed the bottleneck? | Hacker News Reader

Does Lemmy benefit from Rust? Is code execution speed the bottleneck?

programming.dev

53 pointsdragontamer27 comments

There is no good reason for any ordinary web application to hit scaling limits at 1,500 users. The developer vaguely waves his hands at memory issues but it's hard to imagine what kind of reasonably-designed system could possibly be putting memory pressure at that scale.

There's no way any modern language will be your bottleneck at that scale.

I think it's perfectly ordinary for a new web application to hit scaling issues at that number. Accidental O(n^2) stuff, missing database indexes, poor use of memory, excessive I/O are easy mistakes to make during initial implementation and probably will get fixed.

That's what I mean by "no good reason". This is the scale where you start to notice the stupid stuff you let through - N+1's, excessive unnecessary string allocations in inner loops, debug logging etc.

What I mean is, there's nothing fundamental about their tech choices that could explain such low scaling limits. It'll be either poor design/architecture, poor coding standards or both.

I don't think the Lemmy developers were expecting #RedditBlackout to dump 150,000 users onto their federation in a week.

They were probably trucking along with various feature requests when suddenly performance became an issue this past week.

Where are you finding that number, 150,000? The instance in question has one one hundredth that figure and the Lemmy devs claim 27,000 accounts grand total.

Lemmy.world is 31k users alone. Beehaw.org is 12.k users (and each of those users at Beehaw.org were manually curated)

Maybe 150,000 is wrong, I saw it posted somewhere but I'm not sure what the methodology is. But between Lemmy.world and Beehaw.org we're already at 43k+ users.

27k is a gross underestimate.

-------------

EDIT:

https://kbin.social/nodeinfo/2.0

https://fedidb.org/software/lemmy

Lemmy-fediverse is claiming 159,000+ users today.

EDIT2: kbin link must have been glitched. kbin.social's specific nodeinfo is "only" claiming 35k users, which is still substantial.

I agree with your point that whatever the number, they weren’t expecting that surge right now. The actual number is barely relevant.

> Lemmy devs

Programming.dev is _NOT_ the lemmy devs. This is a discussion from https://programmers.dev, a totally separate group of people.

This is just an administrator who is sharing their Lemmy experiences while talking about Rust and such.

Lemmy.ml, the original developer's instance, is well known to be a small server.

Lemmy.world is well into 31,500+ users on a single server and still growing strong. But growing pains have begun to creep in in any case. The server's software clearly needs a few optimization passes.

What does a small server mean here? Is there an architecture diagram for a Lemmy instance somewhere? What parts can't be replicated? Is it a small database server?

Lemmy has become one of the major "lifeboats" during the #RedditBlackout even this past week. This has lead to interesting performance issues and questions... as well as human-level scaling issues.

On the technical side, this https://programming.dev thread (itself, a Lemmy instance) asks how Rust may have affected Lemmy. The ongoing discussion covers a wide variety of topics.

The #1 post right now is frome snowe@programming.dev, the owner of this instance. He took disk-measurements of the server and published them, demonstrating that Lemmy's current bottleneck seems to be RAM (proven by the change in Swapping behavior / Disk-io before-and-after he upgraded RAM).

--------

I do think you can see the benefits of federation in this manner. Users from many instances: https://kbin.social, https://lemmy.world, and https://beehaw.org... and more (all different Lemmy instances) have joined in the discussion and have made a high-quality thread.

---------

Note: I am bullish on federation / Fediverse after playing around with it this past week. I'm less bullish on Lemmy itself, which has proven to be a buggy, laggy mess. Still, there's good discussion happening on Lemmy now (albeit slower due to fewer users). Communities are springing up (and will be dying), and I'm sure instances will spin up and die too. But the idea of federation works, especially for bringing together the discussion.

I think work (and play) needs to happen on Lemmy, kbin, Mastodon, and other Fediverse instances to figure out what is the best software, and best experience for users. Its not quite ready for prime-time, but the core feature set / Minimum-viable-product demo here is extremely exciting to me.

What happens with the content if posted on an instance that shuts down? Will it be kept alive by other instances?

This has already happened in practice, as https://beehaw.org and https://lemmy.world defederated due to some (hopefully temporary) issues.

I don't wanna go into the drama / discussion here but lets just call it some nerd drama + lack of moderation tools that can solve the issue. (Beehaw.org is very confident that if Mastodon-like tools existed, they'd be able to refederate and get the things they want. The issue is being blamed right now on Lemmy's relatively new codebase that's missing some features that Mastodon users are familiar with)

https://Lemmy.world is continues to host the copies of the "old Beehaw.org" material, allowing Lemmy.world users to talk. But the "federation" has shut down, you can only talk with local users (because Beehaw.org serves as the "True copy". So upvotes / downvotes, and federated messages/comments are coordinated by the Beehaw.org server)

In practice, this means that all the Beehaw.org communities has become kinda-sorta useless on https://Lemmy.world. But, the copies of the conversations (up to the point of defederation) do exist and remain on your local (erm... more-local?) server.

It might still remain in the cache of another instance, but there is really no permanence build into the Fediverse. If an instance goes down, its content and user accounts go down too. There is no quick&easy way to migrate content to another host. There are no crypt-ids (e.g. PGP keys), everything is refered to as user@host, even message-ids aren't even globally unique, e.g. these two refer to the same post:

* https://beehaw.org/post/598023

* https://feddit.de/post/881008

Fediverse still has a lot of work to, right now it feels more like a quick hack than something that could significantly improve the Internet in the long run. If an instance goes malicious, you have all the same problems as you'd have with an unfederated server, after all federation is completely optional in the Fediverse.

> Lemmy's current bottleneck seems to be RAM

Rust seems like a sound choice in that regard because it's not garbage collected, so you're not wasting some RAM overhead just to avoid needing to do extra work, and it doesn't have C-style struct packing issues. If you don't specify the layout you need for a user defined type, Rust will assume you are OK with it re-arranging your data structure to be smaller / faster.

If you're willing to put in some work you can often go further with this in Rust than in other "systems programming" languages too. For example C++ Strings have the "Small String optimisation", exactly how this works varies but e.g libc++ (on a modern 64-bit server) String is a 24 byte data structure which can hold up to 22 bytes of textual data inline without yet needing a heap allocation. So if you do a lot of work modifying small amounts of text, Rust's built-in String isn't as efficient as a libc++ String, but the CompactString type from a popular Rust crate is also a 24 byte data structure yet it holds a more impressive 24 bytes of text before needing heap allocation.

Just wanted to pick up on this minor point:

> current bottleneck seems to be RAM

RAM exhaustion is often a secondary side effect of the true underlying bottleneck.

It could be that RAM used for temporarily processing an HTTP request, say, would be freed quicker if the database returned sooner. Maybe the database is the actual bottleneck. And the actual root cause is an indexing / partitioning issue.

It's more like that the bottleneck is any of the following:

    1. Infrastructure
    2. Architecture (not sufficiently decoupled, insufficient cache design, etc.)
    3. Database/storage limitations
    ...
    ...
    99. Rust

There is a mention of WebSockets to each user — could someone verify whether that is true or not? As that would be an absolute overkill, hell I see no reason to use Websockets besides online games, and collab tools where short latency is a requirement. A reddit clone is none of those.

Try logging in programming.dev and check your network tab.

You will see the request being sent as a websocket. The lemmy author implemented lemmy so as to see how fast you can get the site working. Inferno, Rust, WebSockets. He probably didnt have expect the scaling issues that come with it.

Anyways, I am happy the site exists. Kudos! Looking towards better implementation in the next release

You could run a Django Python project, and it will cause an extra 40 ms delay and insane 20-60 ms variance out of thin air due to the framework delay. The question is blatantly stupid.

Historically the bottleneck has been the database IO and then companies value more their human time than technical costs. That's why we have Spring or Python being so awful slow compared with Rust at the backend.

The Techempower benchmarks has the answer:

https://www.techempower.com/benchmarks/

Rust is by far, better than anything else out there.

Django is at 3,2%. Spring at 20%. PHP at 20%. Rust at 99%. JavaScript is at 100% but they're cheating.

For actual OLTP workloads using an async framework is more important than language.

See "multiple queries" tab.

Is ActivityPub not scalable? I keep seeing criticisms of it on this front.

A certain degree of overhead just has to be there, a distributed system needs to exchange information between nodes after all. And both the protocol and many implementations leave room for optimization at least. I think tackling that is one of the long-term challenges for the ecosystem, but I wouldn't say it is categorically "not scalable".

EDIT: As an example, right now in Mastodon federation traffic is handled by sidekiq queues running Ruby. Someone makes a post -> tasks to send that information to all their followers instances get queued, which then send individual HTTP requests to all those instances. This to me feels like something that could be moved into some smaller and more-efficiently-written component that batches requests, makes sure to reuse connections, ... at least for paths that see a lot of traffic.

The way that instances connect to each other is O(n^2), which is not to say it's exactly n^2 but the curve is of that shape. This isn't strictly a problem with ActivityPub as with how it's used, especially in Mastodon. It's why you keep hearing about super-long queues chewing up RAM, and causing long delays - hours or even days - before content appears. It's why Mastodon instances tend to be clusters of larger machines at a scale that you would think can be served by a single smaller machine.

The usual solution for this set of problems is connection and/or request aggregation via a proxy layer. Relays (in fediverse-speak) do exist, but seem very lightly used and do little or no caching because ActivityPub relies heavily on POST (which IMO was probably a bad decision). Any caching that's done has to be ActivityPub-specific. I wrote about this a while ago and almost certainly got some stuff wrong, but it might be interesting nonetheless. At least it drew some interesting comments at the time.

https://gist.github.com/jdarcy/60107fe4e653819138396257df302...

Illustrates how using Rust or any other fast language doesn’t necessarily make things fast.

Reply on news.ycombinator.com